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PREFACE 


This  report  summarizes  the  results  of  a  National  Science  Foundation 
workshop,  Partnerships:  Building  a  New  Foundation  for  Innovation, 
held  June  18-19,  2001,  in  Arlington,  Virginia.  The  workshop  and  this 
report  were  sponsored  by  the  National  Science  Foundation's 
Partnerships  for  Innovation  program  and  were  conducted  within 
RAND's  Science  and  Technology  Policy  Institute. 

ABOUT  THE  S&T  POLICY  INSTITUTE 

Originally  created  by  Congress  in  1991  as  the  Critical  Technologies 
Institute  and  renamed  in  1998,  the  Science  and  Technology  Policy 
Institute  is  a  federally  funded  research  and  development  center 
sponsored  by  the  National  Science  Foundation  and  managed  by 
RAND.  The  Institute's  mission  is  to  help  improve  public  policy  by 
conducting  objective,  independent  research  and  analysis  on  policy 
issues  that  involve  science  and  technology.  To  this  end,  the  Institute 

•  Supports  the  Office  of  Science  and  Technology  Policy  and  other 
Executive  Branch  agencies,  offices,  and  councils; 

•  Helps  science  and  technology  decisionmakers  understand  the 
likely  consequences  of  their  decisions  and  choose  among  alter¬ 
native  policies;  and 

•  Helps  improve  understanding  in  both  the  public  and  private 
sectors  of  the  ways  in  which  science  and  technology  can  better 
serve  national  objectives. 
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In  carrying  out  its  mission,  the  Institute  consults  broadly  with  repre¬ 
sentatives  from  private  industry,  institutions  of  higher  education, 
and  other  nonprofit  institutions. 

Inquiries  regarding  the  Science  and  Technology  Policy  Institute  may 
be  directed  to  the  address  below. 

Helga  E.  Rippen 
Director 

Science  and  Technology  Policy  Institute 
1200  South  Hayes  Street 
Arlington,  VA  22202 
Phone:  (703)  413-1100  ext.  5351 
Email:  stpi@rand.org 
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SUMMARY 


The  National  Science  Foundation  (NSF)  was  created  at  the  onset  of 
the  Cold  War  in  1950  “to  promote  the  progress  of  science;  to  advance 
the  national  health,  prosperity,  and  welfare;  to  secure  the  national 
defense;  and  for  other  purposes.”1  Since  then,  there  have  been 
enormous  changes  in  the  international  and  domestic  arenas,  in  the 
U.S.  economy,  and  in  the  character  of  the  national  innovation  sys¬ 
tem.  Moreover,  the  nation  will  face  substantial  economic  and  social 
challenges  in  the  future  that  differ  in  fundamental  ways  from  those  of 
the  past. 

The  NSF  created  the  Partnerships  for  Innovation  (PFI)  program  in 
2000  as  part  of  an  effort  to  build  a  new  foundation  for  innovation  in 
local  communities  that  is  responsive  to  these  changes  and  emerging 
challenges.  This  effort  is  based  on  partnerships  among  universities, 
industry,  and  local  and  regional  governments.  The  PFI  program's 
goals  are  to: 

•  Catalyze  Partnerships  for  Innovation  that  will  enable  the  trans¬ 
formation  of  knowledge  created  by  the  national  research  and  ed¬ 
ucation  enterprise  into  innovations  that  create  new  wealth;  build 
strong  local,  regional,  and  national  economies;  and  improve  the 
national  well-being; 


^he  NSF’s  continuing  mission  is  set  out  in  the  preamble  to  the  National  Science 
Foundation  Act  of  1950  (Public  Law  810507). 


xii  Building  a  New  Foundation  for  Innovation:  Results  of  a  Workshop 


•  Broaden  the  participation  of  all  types  of  academic  institutions 
and  all  citizens  in  NSF  activities  to  more  fully  meet  the  broad 
workforce  needs  of  the  national  innovation  enterprise;  and 

•  Catalyze  creation  of  the  enabling  infrastructure  necessary  to  fos¬ 
ter  and  sustain  long-term  innovation. 

This  report  summarizes  the  results  of  discussions  that  took  place 
during  a  1-1/2  day  workshop  that  convened  in  June  2001  to  consider 
the  PFI  program  and  the  NSF’s  role  in  the  larger  national  innovation 
enterprise.2  These  discussions  can  be  summarized  broadly  in  terms 
of  what  was  said  about  innovation  and  how  best  to  measure  it,  about 
partnerships  and  how  they  might  be  assessed,  and  about  what  the 
National  Science  Foundation  might  do  to  foster  innovation  gener¬ 
ally,  and  more  specifically,  through  partnerships. 

INNOVATION 

Although  the  NSF  as  yet  has  not  embraced  a  single  definition  for 
innovation,  workshop  participants  generally  used  the  term  in  a  way 
that  focused  on  the  processes  and  mechanisms  for  producing  com¬ 
mercial  applications  of  new  knowledge  rather  than  on  the  products 
or  outputs  from  these  processes.  The  workshop  discussions  implied 
the  following  sort  of  working  definition: 

Innovation  is  the  transformation  of  knowledge  into  products,  pro¬ 
cesses,  systems,  and  services,  with  the  key  elements  of  underlying 
innovation  being:  (1)  knowledge;  (2)  a  skilled  workforce;  and  (3)  in¬ 
frastructure.3 

There  was  broad  support  in  the  workshop  for  the  proposition  that 
innovation  drives  manufacturing  and  other  productivity  growth, 
which  in  turn  drives  economic  growth  and  national  well-being.  The 
PFI  program  seeks  to  connect,  at  the  project  and  programmatic  level, 
knowledge  creation  to  innovation,  and  innovation  to  wealth, 
economic  development,  and,  ultimately,  national  well-being. 


2The  viewpoints  reported  here  were  those  expressed  by  workshop  participants  and  do 
not  necessarily  reflect  the  positions  of  the  National  Science  Foundation  or  RAND. 

3Definition  suggested  by  John  Hurt,  PFI  program  officer. 
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Regarding  the  process  of  transforming  new  knowledge  into  innova¬ 
tion,  a  common  view  was  that  although  there  were  few  proven  for¬ 
mulas  for  innovation,  work  at  the  seams — i.e.,  work  that  cuts  across 
traditional  disciplinary  lines  and  across  university- industry  lines — 
was  where  the  unexpected  could  be  expected  to  happen,  with  the 
result  being  a  more  effective  innovation  process. 

Most  of  the  workshop  discussions  focused  on  the  steps  involved  in 
establishing  partnerships  that  can  bring  an  invention  into  the  com¬ 
mercial  world.  The  commercial  link  in  many  respects  was  viewed  as 
the  most  critical  link.  Characteristics  of  commercially  useful  re¬ 
search  were  seen  to  include  uniqueness  and  responsiveness  to  in¬ 
dustry  and  market  needs.  Simply  identifying  potential  strategic  part¬ 
ners  was  seen  as  perhaps  the  main  challenge,  largely  because  of  im¬ 
perfections  in  information  markets:  Potential  customers  for  inven¬ 
tions  (industry)  and  potential  suppliers  of  inventions  (universities) 
frequently  know  very  little  about  one  another  or  where  they  might 
find  common  cause. 

Another  potential  challenge  was  ensuring  the  requisite  level  of 
compatibility  of  objectives  between  university  and  industry  and  mu¬ 
tual  understanding  of  (and  respect  for)  differences  in  objectives. 
Differences  in  institutional  cultures,  time  horizons,  and  confidential¬ 
ity  needs,  and  the  potential  for  disagreements  over  royalty  shares 
from  intellectual  property,  were  seen  to  be  particularly  problematic. 
Access  to  a  host  of  other  resources— money,  personnel,  and 
infrastructure — also  were  seen  as  potential  challenges.  Attendees 
offered  a  number  of  suggestions  to  overcome  these  challenges,  most 
of  which  aimed  to  improve  communications  in  some  fashion  or 
change  university  policies  that  hindered  the  university’s  ability  to 
contribute  to  innovation. 

The  final  link  in  the  innovation  chain — the  spread  of  benefits  beyond 
the  partnership  and  its  immediate  customers  to  the  broader  local, 
state,  regional,  and  national  economy— received  less  attention.  Nev¬ 
ertheless,  workshop  participants  suggested  that  localities,  regions, 
states,  or  nations  ultimately  compete  with  one  another,  particularly 
in  creating  new  technology  clusters  that  could  lead  to  high-quality 
jobs  and  other  types  of  economic  development,  and  that  competitive 
advantages  were  an  important  consideration  at  this  level.  Neverthe- 
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less,  it  can  be  exceedingly  difficult  to  establish  direct  connections  be¬ 
tween  a  specific  innovation  and  larger  economic  or  societal  effects. 

PARTNERSHIPS 

More  often  than  not,  workshop  participants  described  genuine  part¬ 
nerships  as  dynamic  and  growing  relationships  based  upon  shared 
interests,  trust,  and  an  evolving  technical  relationship.  Participants 
saw  these  relationships  as  necessarily  multifaceted,  including  senior 
researchers,  students,  business  people,  and  others  who  could  find 
common  ground  and  purpose.  Some  argued  that  such  partnerships 
were  inherently  long-term  in  nature,  but  a  vocal  minority  argued  that 
innovative  partnerships  typically  lasted  only  until  the  original  pur¬ 
pose  (commercializing  the  innovation)  was  achieved,  at  which  time 
the  most  innovative  partners  would  move  on  to  the  next  challenge. 

Catalyzing  partnerships  with  these  characteristics  was  seen  to  be 
very  challenging.  Partnerships  required  a  vision  and  performance 
goals  and  benchmarks,  passionate  and  visionary  leaders,  and  part¬ 
ners  who  were  bound  by  an  essential  interdependency  and  shared 
commitment.  Moreover,  to  achieve  success  these  partners  needed  to 
identify  their  competitive  advantages  (and  disadvantages),  resolve 
potential  intellectual  property  (IP)  disputes,  and  develop  and  execute 
strategic  business  plans.  Continued  communication  and  conflict 
resolution  also  were  needed  once  the  commercial  phase  was  under 
way.  In  short,  workshop  participants  warned  that  many  centrifugal 
forces  can  pull  partnerships  apart  and  only  one — the  shared  com¬ 
mitment  and  interdependency  of  the  partners — can  hold  them  to¬ 
gether. 

The  workshop  also  addressed  the  sorts  of  infrastructure  that  can 
sustain  and  nurture  the  spread  of  innovative  activity  over  the  long 
term.  Workshop  deliberations  described  infrastructure  of  three  gen¬ 
eral  kinds:  for  developing  human  capital,  for  developing  networks, 
and  for  providing  direct  support  for  the  innovation  enterprise. 

•  To  educate  and  train  human  capital  for  the  research  enterprise — 
and  the  entrepreneurial  aspects  of  innovation— workshop  partic¬ 
ipants  saw  the  requirement  for  an  innovative  educational  pro¬ 
cess  whose  most  important  goal  was  to  serve  the  student,  to 
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provide  hands-on  education,  and  then  to  recruit  and  retain  the 
best  young  scientists  and  engineers. 

•  The  intellectual  capital  and  know-how  embodied  in  young  sci¬ 
entists  and  engineers,  honed  through  advanced  education  and 
training,  is  embedded  in  social  networks  characterized  by  shared 
commitment  and  trust.  Such  networks  can  be  built  through  ex¬ 
tended  interactions  and  problemsolving,  and  represent  a  form 
of  social  capital  that,  in  some  workshop  participants’  minds, 
seemed  to  be  the  most  important  type  of  infrastructure  of  all. 

•  Finally,  a  base  of  operational  support  is  essential,  because  sus¬ 
tainable  partnerships  cannot  exist  without  such  support.  Ulti¬ 
mately,  this  was  viewed  both  as  a  resource  issue — a  diversified 
base  of  private  investment — and  as  a  physical  place  that  can 
provide  a  context  for  incubation;  technical,  management,  and 
administrative  support;  laboratory  and  other  capacity;  commu¬ 
nications  services;  and  reliable  sources  of  capital.  The  impor¬ 
tance  of  support  for  federal  innovation  programs  from  at  least 
one  key  constituency — the  Congress — also  was  seen  as  critical. 

A  final  aim  of  the  PFI  program  is  to  broaden  the  participation  of  all 
institutions  and  people  in  the  innovation  enterprise,  including  un¬ 
derrepresented  and  underserved  institutions  and  individuals.  Work¬ 
shop  participants  seemed  to  agree  that  this  can  successfully  be  ac¬ 
complished  through  a  solicitation  and  selection  process  that  rewards 
quality;  none  argued  that  by  including  such  institutions  technical 
standards  would  necessarily  suffer. 

IMPLICATIONS  FOR  THE  NSF 

Two  major  points  affecting  both  the  PFI  program  and  NSF  came  out 
of  the  workshop. 

First,  there  was  nearly  unanimous  support  for  a  formal  evaluation  of 
the  PFI  program  by  an  independent,  paid  evaluator.  Such  an  effort 
was  viewed  as  being  consistent  both  with  the  NSF’s  general  com¬ 
mitment  to  evaluation  as  an  aid  to  outcome-based  management  and 
with  its  specific  obligations  under  the  Government  Performance  and 
Results  Act  of  1993  (GPRA) .  According  to  participants,  such  an  eval¬ 
uation  should  focus  on  the  most  critical  questions  related  to  innova- 
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tion  and  the  partnerships  and  should  provide  outcome  and  process 
measures  for  individual  projects  and  the  program  as  a  whole.4 

Second,  the  workshop  endorsed  both  an  expanded  NSF  role  in  pro¬ 
moting  innovation  and  partnerships  through  the  PFI  program  and 
continued  efforts  by  the  NSF  to  further  diversify  and  better  exploit 
synergisms  in  its  support  for  innovation.  The  presentations  and 
workshop  discussions  evidenced  substantial  enthusiasm  for  the  PFI 
and  other  NSF  programs  that  support  innovation  and  university- 
industry  collaborative  efforts,  without  favoring  any  particular  model 
(e.g.,  PFI)  over  any  other  (e.g.,  Industry/ University  Cooperative 
Research  Centers,  or  I/UCRCs). 

However,  the  workshop  left  unaddressed  a  number  of  critically  im¬ 
portant  strategic-level  questions  that  the  NSF  should  consider  in 
current  and  future  planning:  How  can  the  NSF  refine  its 
understanding  of  the  relative  effectiveness  of  its  innovation 
programs  in  promoting  its  objectives?  Which  of  the  available 
programs  and  models  is  the  most  appropriate  tool  for  the  NSF  to  use 
under  which  circumstances?  In  what  ways  can  (or  should)  the  NSF 
exploit  synergies  between  programs  (e.g.,  partnerships  that  are 
incubated  in  an  engineering  research  center  (ERC)  or  I/UCRC)?  In 
what  ways  can  (or  should)  this  mosaic  of  programs  be  considered 
together  as  a  larger  whole  to  ensure  that  the  NSF’s  enterprise-wide 
portfolio  of  innovation- catalyzing  programs  matches  its  strategic 
intent  and  its  presumed  desire  to  achieve  an  optimal  program  mix 
and  level  of  diversification?  What  is  the  best  balance  or  tradeoff 
among  the  various  potentially  conflicting  imperatives  (e.g., 
education  and  workforce  development,  academic  research, 
innovation,  and  diversity)?  What  factors  are  associated  with  the 
success  or  failure  of  technology  partnerships,  networks,  and  clusters? 

Insofar  as  these  questions  seems  to  be  at  the  heart  of  the  NSF's  effort 
to  build  a  new  foundation  for  innovation  in  the  new  century  while  re¬ 
maining  true  to  its  historical  purpose,  they  are  particularly  deserving 
of  further  analysis,  discussion,  and  debate. 


4One  reviewer  of  this  report  suggested  that  another  way  to  categorize  the  metrics  for 
evaluation  would  be  in  terms  of  input,  output,  outcome,  and  in-process  metrics. 
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Chapter  One 

INTRODUCTION 


The  National  Science  Foundation  (NSF)  was  created  at  the  onset  of 
the  Cold  War  in  1950  “to  promote  the  progress  of  science;  to  advance 
the  national  health,  prosperity,  and  welfare;  to  secure  the  national 
defense;  and  for  other  purposes.”1  Consistent  with  this  mission,  the 
NSF  is  authorized  by  law  to  initiate  and  support  both  basic  scientific 
research  and  applied  research,  programs  to  strengthen  scientific  and 
engineering  research  potential,  science  and  engineering  education 
programs  at  all  levels,  and  an  information  base  for  science  and  engi¬ 
neering  appropriate  for  development  of  national  and  international 
policy.2 

Much  has  changed  since  the  NSF’s  founding.  There  have  been 
enormous  changes  in  the  international  arena,  as  the  imperative  of 
military  competition  has  largely  been  supplanted  by  that  of  eco¬ 
nomic  competition.  So  too  have  changes  occurred  in  the  domestic 
arena,  as  a  decade  or  more  of  “creative  destruction”  has  transformed 
the  U.S.  manufacturing  and  service  components  of  the  economy.3 
And  the  challenges  ahead— among  them  ensuring  a  workforce  that  is 
adequate  in  size,  composition,  and  skills  to  support  the  level  of  eco¬ 
nomic  growth  and  vitality  that  will  be  necessary  as  the  ranks  of  re¬ 
tirees  swell  with  the  retirement  of  the  baby  boomers — are  formidable 
ones.  Last,  the  nature  of  the  innovation  enterprise  also  continues  to 


^he  NSF’s  continuing  mission  is  set  out  in  the  preamble  to  the  National  Science 
Foundation  Act  of  1950  (Public  Law  810507). 

2National  Science  Foundation  (2001),  p.  1. 

3The  phrase  “creative  destruction"  is  credited  to  the  economist  Joseph  A.  Schumpeter. 
For  an  example  of  the  use  of  the  concept  in  the  area  of  innovation,  see  Stein  (1997) . 
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change,  with  the  result  that  industry  shoulders  an  increasingly  large 
share  of  the  burden  for  research  and  development  (R&D),  and  the 
federal  government's  share  has  shrunk.4 

For  the  NSF,  these  changes  have  motivated  an  increased  focus  on 
ideas,  innovation,  and  managing  knowledge  as  a  critical  resource — 
the  drivers  of  productivity  and  economic  growth.5  Moreover, 
changes  in  the  innovation  process  itself  call  for  a  different  set  of  tools 
and  institutional  arrangements  to  be  effective. 

In  recognition  of  these  changes,  the  NSF  recently  developed  a 
vision— “enabling  the  nation’s  future  through  discovery,  learning, 
and  innovation” —  to  supplement  its  longstanding  statutory  mission 
and  to  highlight  the  centrality  of  innovation  in  that  mission.6 

NSF  support  for  innovation  is  not  new:  Traditionally,  the  NSF’s  role 
in  the  national  innovation  system  has  been  to  provide  funding  for 
basic  research  to  gain  new  knowledge,  education,  and  training  for 
the  current  and  future  workforce,  and  infrastructure  that  serves  as 
the  foundation  for  research.  The  NSF  likes  to  say  that  it  supports 
people  (education  and  workforce  development),  ideas  (gaining 
knowledge  that  can  lead  to  innovations),  and  tools  (infrastructure).7 
It  does  so  by  supporting  the  development  of  intellectual  capital, 
integrating  research  and  education,  and  promoting  partnerships. 

Of  the  three  main  types  of  research  and  development  collabora¬ 
tion — industry-led  consortia,  collaborations  between  universities 
and  industry,  and  collaborations  between  industry  and  federal  labo¬ 
ratories — the  NSF’s  mission  statement  dictates  a  focus  on  universi¬ 
ties  by  virtue  of  their  role  as  centers  of  research,  development,  and 
training.  The  NSF  always  has  had  a  close  association  with  universi¬ 
ties  and  has  traditionally  been  able  to  achieve  many  of  its  goals 
through  these  institutions.  As  the  NSF's  goals  have  evolved  to  meet 
the  nation’s  changing  needs — including  those  of  the  national  inno- 


4See  for  example,  Hicks  et  al.  (2001). 

5Stein  (1997). 

5The  importance  of  basic  research  on  a  use-inspired  basis  and  the  view  that  the  linear 
R&D  model  is  no  longer  valid  are  discussed  in  Stokes  (1997). 

7From  Bordogna  (2000). 
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vation  system  and  broader  economy — NSF  has  sought  to  develoj 
new  ways  to  achieve  its  goals  through  universities.8 

Although  the  NSF  historically  has  supported  collaborative  researcl 
programs  that  linked  universities,  industry,  and  governments,  these 
efforts  were  greatly  expanded  beginning  with  the  university-industry 
Cooperative  Research  program  in  the  1970s,  during  the  1980s  wit! 
the  creation  of  Engineering  Research  Centers  (ERCs),  and  in  tht 
1990s  with  the  Grant  Opportunities  for  Academic  Liaison  with 
Industry  (GOALI)  program.  The  NSF’s  model  of  university- industry 
partnerships  has  been  widely  emulated  by  other  federal  agencies.9 

The  NSF  today  supports  more  than  100  university  centers,  including 
Industry/ University  Cooperative  Research  Centers  (I/UCRCs),  State- 
I/UCRCs,  Engineering  Research  Centers,  Materials  Research  Science 
and  Engineering  Centers,  Science  and  Technology  Centers,  and  Mi¬ 
nority  Research  Centers  of  Excellence,  as  well  as  GOALI,  supporting 
links  between  university  students  and  their  counterparts  in  industry, 
and  Experimental  Program  to  Stimulate  Competitive  Research 
(EPSCoR).  Total  NSF  activity  involving  industry  was  estimated  in 
1999  to  be  about  $1  billion,  nearly  a  quarter  of  the  NSF's  overall 
budget.10 

Smaller  firms  and  institutions  have  become  increasingly  important 
to  the  national  innovation  system  over  the  last  decade,11  necessitat¬ 
ing  a  greater  focus  on  innovation  at  the  micro-level.  Meanwhile,  ad¬ 
verse  demographic  trends  that  may  result  in  a  smaller  future  work¬ 
force  and  a  larger  population  of  pensioners  demands  investments 
that  can  continue  to  yield  productivity  increases  and  economic 


8This  discussion  focuses  on  NSF’s  role  in  innovation  and  partnerships,  but  a  broad 
literature  exists  on  these  topics.  For  additional  reading,  see  American  Council  on  Ed¬ 
ucation  (2001)  and  Government-University-Industry  Research  Roundtable  (1999). 

9Mowery  (1998). 

10De  Graaf  (1999).  There  are  currently  25  NSF  Science  and  Technology  Centers. 

^For  example,  between  1989  and  1997,  the  percentage  of  industrial  research  and  de¬ 
velopment  performed  by  firms  with  fewer  than  500  employees  increased  from  7.7  to 
15.5  percent,  while  the  percentage  performed  by  firms  of  25,000  or  more  employees 
fell  from  76  to  66  percent.  See  National  Science  Board  (2000),  Vol.  2,  Table  2-53,  p.  A- 
95.  Between  1992  and  1999,  the  percentage  of  academic  research  and  development 
performed  by  the  top  10  institutions  fell  from  19.3  to  18.1  percent  of  all  academic  R&D. 
National  Science  Foundation  (1999). 
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growth.  These  productivity  increases  will  in  turn  require  support  for 
innovation  in  traditionally  underrepresented  institutions  and  sup¬ 
port  for  young  scientists  and  engineers  from  traditionally  underrep¬ 
resented  segments  of  the  population. 

Specifically,  several  regions  around  the  country  have  primarily  de¬ 
pended  upon  agricultural  and  natural  resources  for  economic  well¬ 
being.  These  regions  are  recognizing  that  they  can  no  longer  be 
solely  dependent  upon  these  resources  and  need  to  look  elsewhere 
for  economic  growth.  Universities  provide  the  type  of  opportunities 
to  generate  knowledge  and  wealth  for  local  and  region^  communi¬ 
ties  through  education  and  training  of  the  current  and  future  work¬ 
force,  infrastructure,  research  and  development,  and  technology 
transfer.12  Members  of  these  communities  (from  government  to  in¬ 
dustry)  can  tap  the  resources  of  universities  (and  vice  versa)  through 
partnerships. 

In  creating  the  Partnerships  for  Innovation  (PFI)  program,  the  NSF 
was  truly  recognizing  for  the  first  time  both  changes  in  the  national 
innovation  system  and  the  possible  economic  and  social  challenges 
the  nation  will  face  in  the  future.  Scientific  research  does  drive 
technological  innovation,  but  the  reverse  also  occurs.  Innovation 
depends  upon  a  mutual  synergistic  set  of  interactions  that  not  only 
include  science,  engineering,  and  technology  but  social,  political, 
and  economic  interactions  as  well. 

The  PFI  program  responds  to  these  challenges  by  supporting  select 
university-industry-government  partnerships  with  grants  that  aim  to 
create  a  sustainable  basis  for  future  innovation.  The  PFI  was  a  first - 
of-its-kind  program  for  the  National  Science  Foundation  due  to  its 
major  emphasis  on  innovation.  These  partnerships  promote  a  wide 
range  of  goals,  including  basic  research,  workforce  education  and 
training,  development  of  infrastructure,  technology  transfer,  and 
commercialization  for  the  purpose  of  innovation.  PFI  resides  under 
the  NSF’s  Directorate  for  Education  and  Human  Resources,  with  the 
principal  aims  of  enhancing  the  connections  between  new  knowl¬ 
edge  created  in  the  discovery  process  and  learning  and  innovation. 
Concurrently,  the  PFI  program  addresses  the  NSF’s  strategic 


12A  known  example  is  Research  Triangle  Park,  in  which  educational  institutions  were 
tapped  to  bring  economic  growth  to  the  region. 
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intention  to  broaden  the  participation  of  people  and  institutions  in 
NSF  activities. 

The  PFI  program  helps  to  build  creative  interactions  in  local  com¬ 
munities  between  colleges  and  universities,  government  agencies, 
foundations,  and  private  corporations.  Each  partnership  is  tailored 
to  help  a  community  better  position  itself  to  accommodate  educa¬ 
tion,  research,  and  development  at  the  local  and  regional  level,  with 
the  goal  of  advancing  local  social  and  economic  opportunities  and 
well-being.13 

The  PFI  program's  goals  are  to: 

•  Catalyze  the  PFI  program  to  enable  the  transformation  of  knowl¬ 
edge  created  by  the  national  research  and  education  enterprise 
into  innovations  that  create  new  wealth;  build  strong  local,  re¬ 
gional  and  national  economies;  and  improve  the  national  well¬ 
being; 

•  Broaden  the  participation  of  all  types  of  academic  institutions 
and  all  citizens  in  NSF  activities  to  more  fully  meet  the  broad 
workforce  needs  of  the  national  innovation  enterprise;  and 

•  Catalyze  creation  of  the  enabling  infrastructure  necessary  to  fos¬ 
ter  and  sustain  long-term  innovation.14 

This  report  summarizes  the  results  of  a  1-1/2  day  workshop  held  in 
lune  2001  that  was  convened  to  discuss  university- industry  partner¬ 
ships  and  more  specifically  the  PFI  program.15  The  objectives  of  the 
workshop  were  threefold:  The  first  was  to  identify  the  major  chal¬ 
lenges  that  PFI  grantees  might  face  in  FY  2001  and  to  identify  alter¬ 
natives  available  to  meet  these  challenges.  The  second  was  to  pro¬ 
vide  the  NSF  with  insights  into  innovation  and  partnerships  that 
might  improve  the  PFI  program  in  successive  years.  Finally,  the 
workshop  aimed  to  identify  broader  issues,  challenges,  and 


13National  Science  Foundation  (2000). 

14For  general  information  about  PFI,  see  http://www.ehr.nsf.gov/pfi/. 

15Workshop  proceedings  can  be  found  at  http://www.rand.org/scitech/stpi/ 
Partnerships/index.html. 
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opportunities  the  NSF  should  consider  in  promoting  university- 
industry  and  other  partnerships  in  programs  such  as  the  PFI. 

To  paraphrase  one  breakout  session:  “We  need  to  measure  innova¬ 
tion,  and  partnerships.”  Accordingly,  following  Chapter  Two,  which 
provides  an  overview  of  the  PFI  program  and  the  June  2001 
workshop,  Chapters  Three  and  Four  report  workshop  discussions 
related  to  these  two  topics  and  their  measurement,  and  most  of 
Chapter  Five  summarizes  workshop  recommendations  regarding  an 
evaluation  of  the  PFI  program  and  the  NSF’s  role  in  supporting 
innovation.  A  number  of  appendixes  also  are  provided  containing 
relevant  background  information  on  the  workshop  and  the 
partnerships. 

Throughout  most  of  this  material— Chapter  Two  through  most  of 
Chapter  Five— our  aim  has  been  to  faithfully  report,  summarize,  and 
synthesize,  without  critiquing  or  implying  endorsement  by  RAND  or 
the  National  Science  Foundation,  the  viewpoints  expressed  by  work¬ 
shop  participants.  We  relied  on  the  notes  of  rapporteurs  in  each 
breakout  session,  plenary  presentations  by  breakout  session  chairs, 
and  a  transcript  of  the  plenary  sessions  as  our  main  source  material. 
We  restructured  this  material  thematically  and  provided  connecting 
threads — or  occasional  references  to  other  material — to  improve 
context  or  flow,  or  where  it  otherwise  seemed  warranted.  At  the  end 
of  Chapter  Five  we  share  our  own  views  but  only  in  the  limited  con¬ 
text  of  identifying  what  we  feel  to  be  the  main  questions  that  the  NSF 
will  need  to  address  as  it  considers  its  future  role  in  catalyzing 
innovation. 


Chapter  Two 

OVERVIEW  OF  THE  PFI  PROGRAM  AND  THE 

WORKSHOP 


The  Partnerships  for  Innovation  program  was  created  in  FY  2000  as  a 
result  of  a  Congressional  appropriation  of  $8.5  million  to  initiate  a 
new  innovation  partnership  effort.1  On  March  10,  2000,  a  PFI  plan¬ 
ning  workshop  was  held  to  develop  a  shared  understanding  of  the 
concepts  of  “innovation"  and  “partnerships  for  innovation"  includ¬ 
ing  specific  examples;  to  provide  guidance  for  the  PFI  solicitation; 
and  to  identify  future  actions,  such  as  the  workshop  held  in  June 
2001,  to  advance  the  PFI  initiative.2  The  findings  from  the  planning 
workshop,  in  addition  to  input  from  NSF  staff,  led  to  the  creation  of 
the  PFI  program  solicitation  shortly  thereafter. 

THE  PROGRAM  SOLICITATION 

The  program  solicitation  called  for  proposals  for  partnerships  among 
universities,  government,  and  the  private  sector  that  would  explore 
new  approaches  to  support  and  sustain  innovation  in  the  long  term. 
Degree-granting  academic  institutions  of  higher  learning  were  to 
serve  as  lead  institutions  or  partners,  in  that  they  were  responsible 
for  the  overall  management  of  the  proposed  partnership.  At  a  mini¬ 
mum,  the  partnership  was  to  include  an  academic  institution  and  a 
private  sector  organization.  In  addition,  other  academic  institutions, 
private  sector  firms  (including  entrepreneurs  and  venture  capital- 


^ee  NSF  Section  of  Public  Law  106-74,  VA/HUD  and  Other  Independent  Agencies  Act 
for  FY  2000  at  http://www.nsf.gov/od/lpa/congress/106/106majorleg.htm. 

2From  Bordogna  (2000). 
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ists),  state/local  government  entities,  and  nonprofit/trade/profes- 
sional  associations  could  be  involved. 

Eligibility  limitations  were  such  that  degree-granting  academic  insti¬ 
tutions  of  higher  learning  could  participate  in  no  more  than  two 
partnership  proposals  and  could  serve  as  the  lead  for  only  one.  A 
senior  institutional  administrator  (dean  or  higher)  in  the  lead  institu¬ 
tion  was  to  serve  as  the  co-  or  principal  investigator  of  the  proposal. 
The  proposed  partnerships  were  to  request  total  budgets  ranging 
from  $300,000  to  $600,000  for  the  award's  duration  of  two  to  three 
years. 

THE  REVIEW  PROCESS 

A  total  of  130  proposals  were  submitted  for  consideration  in  the  first 
year  (2000).  The  NSF  solicited  reviews  of  these  proposals  from  peers 
with  expertise  in  the  substantive  area  of  the  proposed  project,  with 
the  reviewers  selected  by  program  officers  charged  with  the  oversight 
of  the  review  process.  The  NSF  invited  each  applicant  to  suggest,  at 
the  time  of  submission,  the  names  of  appropriate  or  inappropriate 
reviewers.  Care  was  taken  to  ensure  that  reviewers  had  no  conflicts 
with  the  proposer.  Special  efforts  were  made  to  recruit  reviewers 
from  nonacademic  institutions,  minority- serving  institutions,  or  ad¬ 
jacent  disciplines  to  that  principally  addressed  in  the  proposal. 

Proposals  were  reviewed  against  the  following  general  review  criteria 
established  by  the  National  Science  Board: 

•  What  is  the  intellectual  merit  of  the  proposed  activity?  How  im¬ 
portant  is  the  proposed  activity  to  advancing  knowledge  and  un¬ 
derstanding  within  its  own  field  or  across  different  fields?  How 
well  qualified  is  the  proposer  (individual  or  team)  to  conduct  the 
project?  To  what  extent  does  the  proposed  activity  suggest  and 
explore  creative  and  original  concepts?  How  well  conceived  and 
organized  is  the  proposed  activity?  Is  there  sufficient  access  to 
resources? 

•  What  are  the  broader  effects  of  the  proposed  activity?  How  well 
does  the  activity  advance  discovery  and  understanding  while 
promoting  teaching,  training,  and  learning?  How  well  does  the 
proposed  activity  broaden  the  participation  of  underrepresented 
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groups  (e.g.,  women,  minorities,  the  disabled,  or  those  in 
particular  geographic  areas)?  To  what  extent  will  it  enhance  the 
infrastructure  for  research  and  education,  such  as  facilities, 
instrumentation,  networks,  and  partnerships?  Will  the  results  be 
disseminated  broadly  to  enhance  scientific  and  technological 
understanding?  What  will  the  benefits  of  the  proposed  activity  to 
society  be? 

Principal  investigators  also  were  asked  to  address  the  following  ele¬ 
ments  in  their  proposal  to  provide  reviewers  with  the  information 

necessary  to  respond  fully  to  both  of  the  above-described  NSF  merit 

review  criteria: 

•  Integration  of  Research  and  Education.  One  principal  strategy  in 
support  of  the  NSF’s  goals  is  to  foster  integration  of  research  and 
education  through  the  programs,  projects,  and  activities  it  sup¬ 
ports  at  academic  and  research  institutions.  These  institutions 
provide  abundant  opportunities  where  individuals  may  concur¬ 
rently  assume  responsibilities  as  researchers,  educators,  and 
students  and  where  all  can  engage  in  joint  efforts  that  infuse  ed¬ 
ucation  with  the  excitement  of  discovery  and  enrich  research 
through  the  diversity  of  learning  perspectives. 

•  Integrating  Diversity  into  NSF  Programs,  Projects,  and  Activities. 
Broadening  opportunities  and  enabling  the  participation  of  all 
citizens — including  women,  underrepresented  minorities,  and 
persons  with  disabilities — are  essential  to  the  health  and  vitality 
of  science  and  engineering.  The  NSF  is  committed  to  this  prin¬ 
ciple  of  diversity  and  deems  it  central  to  the  programs,  projects, 
and  activities  it  considers  and  supports. 

The  following  additional  PFI -specific  review  criteria  also  were  used: 

•  Responsiveness  of  the  proposal  to  the  goals  of  the  PFI  program; 

•  The  degree  to  which  the  proposed  activity  will  stimulate  new  in¬ 
novation  opportunities  for  the  partner  organizations; 

•  The  potential  effect  of  the  proposed  innovation  activities  on  the 
economic  or  societal  well-being  of  the  region, 

•  Potential  of  the  proposed  partnership  to  foster  and  sustain  in¬ 
novation  in  the  long  term; 
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•  The  degree  to  which  institutions  that  serve  groups  currently  un¬ 
derrepresented  in  the  science,  engineering,  and  technological 
workforce  participate  in  the  proposed  innovation  activity;  and 

•  The  degree  to  which  institutions  that  serve  regions  or  sectors  not 
yet  fully  participating  in  the  innovation  enterprise  contribute  to 
the  proposed  activities. 

In  addition  to  these  criteria,  in  making  the  final  award  decisions  the 
NSF  also  considered  geographic  distribution  and  diversity  of  lead 
institutions,  the  likely  distribution  of  societal  effects,  and  the 
distribution  of  technology  or  industry  sectors  served. 

All  proposals  were  reviewed  by  at  least  three  reviewers  outside  the 
NSF  who  were  experts  in  the  particular  field  represented  by  the 
proposal.  Reviewers  were  asked  to  formulate  a  recommendation  to 
either  support  or  decline  each  proposal.  The  program  officer 
assigned  to  manage  the  proposal’s  review  considered  the  advice  of 
reviewers  and  formulated  a  recommendation. 


THE  PARTNERSHIPS 

Of  the  130  proposals  submitted  in  the  first  year,  the  PFI  program 
awarded  grants  to  24  promising  partnerships  that  aimed  to  translate 
knowledge  gained  from  basic  research  into  new  products, 
businesses,  and  services;  to  provide  workforce  education  and 
training  opportunities  focused  on  innovation;  and  to  develop 
infrastructure  that  would  support  future  innovation.3 

To  provide  a  better  sense  of  the  composition  of  these  partnerships, 
Table  2. 1  shows  that  all  24  university  awardees  partnered  with  indus¬ 
try,  although  some  also  partnered  with  others  from  universities,  or 
from  government,  from  the  venture  capital  community,  or  from  in¬ 
cubators.  And  Table  2.2  reports  that  with  20  partnerships  citing  it  as 
a  major  goal,  technology  transfer  was  by  far  the  most  frequently 
identified  major  goal  in  these  partnerships,  followed  by  education 


3National  Science  Foundation  (2000).  Descriptions  of  these  partnerships  can  be 
found  in  Appendix  B.  In  its  second  year  (2001),  109  proposals  were  submitted;  the 
NSF  recommended  funding  for  12  partnerships. 
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Table  2.1 

Partners  of  the  24  University  Awardees 


Institution 

Number 

Private  sector 

24 

University 

19 

Government 

16 

Venture  Capital 

5 

Incubators 

4 

Table  2.2 

Goals  for  the  24  Partnerships 


Goals 

Major 

Secondary 

Technology  transfer 

20 

2 

Education 

8 

3 

Infrastructure 

3 

9 

and  development  of  infrastructure;  infrastructure  development  was 
the  most  often  mentioned  secondary  goal,  followed  by  education 
and  technology  transfer. 

THE  WORKSHOP 

The  PFI  workshop  included  a  mix  of  formal  presentations,  panel  dis¬ 
cussions,  breakout  sessions  to  address  specific  topical  areas,  and 
plenary  sessions  where  the  fruits  of  these  discussions  could  be 
shared  and  further  discussed.4  Discussants  included  both  members 
of  the  PFI’s  steering  group  and  PFI  grantees,  as  well  as  individuals, 
primarily  from  universities,  who  are  interested  in  the  topic  of  inno¬ 
vation.5 

In  the  next  two  chapters,  we  report  the  viewpoints,  discussions,  and 
conclusions  expressed  by  workshop  participants  regarding  innova¬ 
tion  (Chapter  Three)  and  sustainable  partnerships  (Chapter  Four). 
What  follows  are  not,  in  the  typical  sense  of  the  term,  the  proceed- 


4See  Appendix  D  for  the  workshop’s  agenda  and  Appendix  E  for  the  questions  that 
were  put  to  each  breakout  session. 

5Appendix  C  provides  a  complete  list  of  workshop  attendees. 
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ings  from  the  workshop;  rather,  we  generally  have  sought  to  organize 
the  viewpoints  of  workshop  participants  according  to  broad  themes; 
in  some  cases,  modest  efforts  also  were  made  to  add  some  context. 


Chapter  Three 

INNOVATION:  FROM  THE  CREATION  OF  KNOWLEDGE 

TO  NATIONAL  WELL-BEING 


[IJnnovation  is  a  locally  driven  process  that  succeeds  where  organi¬ 
zational  conditions  foster  the  transformation  of  knowledge  into 
products,  processes,  systems,  and  services.  The  local  part  of  this  is 
very  important . . .  it’s  only  when  local  expertise,  and  knowledge  of 
local  needs,  local  conditions,  and  local  resources,  can  be  brought  to 
the  picture,  that  innovation  will  truly  happen. 


Mariann  (Sam)  Jelinek,  Program  Director,  Innovation  and 
Organizational  Change  Program,  National  Science  Foundation 

Although  the  NSF  has  not  yet  embraced  a  single  definition  for 
innovation,  workshop  participants  generally  used  the  term  in  a  way 
that  suggested  a  focus  on  the  processes  and  mechanisms  for 
producing  commercial  applications  of  new  knowledge  rather  than 
on  the  products  or  outputs  from  these  processes.1  In  fact,  there 
seemed  to  be  broad  support  in  the  workshop  for  the  proposition  that 
innovation  drives  manufacturing  and  other  productivity  growth, 
which  in  turn  drives  economic  growth  and  national  well-being.2 

As  suggested  by  its  goals,  the  PFI  program  seeks  to  connect,  at  the 
project  and  programmatic  level,  knowledge  to  innovation  and 
innovation  to  wealth,  economic  development,  and,  ultimately, 
national  well-being.  This  chapter  summarizes  workshop  discussions 


^ee  Popper  and  Wagner  (2001),  p.  1. 
2See  Milbergs  (2001). 
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regarding  each  of  these  links  in  this  innovation  chain  and  reports  the 
metrics  that  were  suggested  by  workshop  participants  and  that  may 
help  in  measuring  key  outcomes  and  processes  at  the  project  and 
program  level. 

FROM  KNOWLEDGE  TO  INNOVATION 

The  first  link  in  the  innovation  chain  involves  transforming  what  may 
be  predominantly  theoretical  knowledge  into  a  practical  application, 
whether  in  terms  of  a  product  or  a  process.3  The  tangible  outputs 
include  both  a  technology  that  can  be  transferred  to  industry  and  the 
knowledge  capital  (know-how)  embodied  in  scientists  and  engineers 
who  are  responsible  for  technology  breakthroughs;  both  can  dissem¬ 
inate  through  the  economy.4 

Mining  the  Seams 

A  common  view  that  was  expressed  was  that  work  at  the  seams — i.e., 
work  that  cuts  across  traditional  disciplinary  lines  and  across 
university- industry  lines — was  where  the  unexpected  could  be  ex¬ 
pected  to  happen,  with  the  result  being  innovation.5  The  reason 
seemed  to  be  that  solutions  not  obvious  in  one  realm  might  be  per¬ 
fectly  obvious  in  another  and  that  these  sorts  of  interactions  en¬ 
hanced  the  prospects  that  such  solutions  could  be  made  obvious.6 


3Edwin  Mansfield  examined  the  extent  to  which  technological  innovations  in  various 
industries  have  been  based  on  recent  academic  research.  See  Mansfield  (1991a). 

4Richard  Florida  (1999)  has  suggested  that  the  principal  output  of  the  innovation  sys¬ 
tem  is  knowledge  workers  who  have  know-how,  not  technologies  to  be  transferred. 
See  Florida  (1999). 

5The  PFI's  focus  on  the  seams  was  a  feature  of  the  program  that  most  workshop 
participants  seemed  to  find  quite  laudable. 

6  A  recent  study  by  Kaufmann  and  Todtling  concludes  that  "[c]rossing  the  border  to 
science,  in  particular,  increases  the  diversity  of  firms'  innovation  partners  and  respec¬ 
tive  innovation  stimuli  which,  in  turn,  improves  the  capability  of  firms  to  introduce 
more  advanced  innovations.”  See  Kaufmann  and  Todtling  (2001). 
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Measurement  Issues 

The  remainder  of  this  chapter  and  the  next  describe  some  of  the 
measurement  issues  that  arose  in  the  context  of  the  discussions  that 
were  just  summarized,  in  this  particular  case,  the  process  by  which 
new  knowledge  can  yield  innovation.  We  organize  this  discussion 
around  two  measures:  outcome  measures  that  describe  ultimate 
success  or  failure  and  “signposts” — process  measures  that  may  be 
useful  indicators  that  the  innovation  process  is  on  track  or,  alterna¬ 
tively,  can  serve  as  early  warning  indicators  that  something  may  be 
wrong.  We  also  distinguish  between  project-level  measures  that  re¬ 
late  to  the  performance  of  the  individual  partnerships  and  program - 
level  measures  that  are  generally  summative  and  relate  to  the  overall 
PFI  program.7 

Outcome  Measures.  The  principal  project-level  outcomes  of  interest 
in  this  link  of  the  innovation  chain  were  said  to  be  the  transfor¬ 
mation  of  new  knowledge  into  commercial  applications.  A  number 
of  suggested  project-level  outcome  measures  related  to  innovation 
and  answered  the  fundamental  question,  “Has  the  project  con¬ 
tributed  to  innovation?”  Among  those  identified  by  workshop  dis¬ 
cussants  as  being  reasonable  indicators  of  successful  innovation 
were  the  number  of  research  papers  reporting  success,  inventions, 
disclosures,  and  patent  filings  and  awards.8’  Additionally,  the  project 
might  be  measured  on  the  basis  of  its  dissemination  of  innovative 
work,  whether  through  journals  (including  economic  and  trade  jour¬ 
nals),  conferences,  or  other  means. 


7A  fuller  description  of  this  framework  is  provided  in  Appendix  G,  which  describes 
design  considerations  for  an  evaluation  of  the  PFI  program.  One  reviewer  of  this  re¬ 
port  suggested  that  another  way  to  categorize  the  metrics  for  evaluation  would  be  in 
terms  of  input,  output,  outcome,  and  in-process  metrics.  In  the  creation  and  imple¬ 
mentation  of  a  set  of  metrics,  the  performance  may  be  driven  in  unintended  ways  as  a 
result  of  certain  biases.  The  metrics  discussed  in  this  report  are  primarily  based  on  the 
discussions  at  the  workshop  and  do  not  give  a  complete  picture  of  the  topic.  For  ad¬ 
ditional  reading  on  biases/problems  of  metrics,  see  Popper  (1995). 

8The  development  of  prototypes  or  proofs  of  concept  was  not  explicitly  mentioned.  A 
peer  reviewer  for  this  report  suggested  that  the  these  indicators  were  output  rather 
than  outcome  measures. 
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More  broadly,  however,  attendees  saw  the  program-level  question  of 
whether  the  PFI  program  was  catalyzing  partnerships  as  a  critically 
important  one  and  wondered  what  sorts  of  counterfactual  analyses 
and  systematic  measures  were  needed  to  properly  assess  this 
question;9  there  were  few  firm  conclusions. 

Signposts.  Attendees  described  process  measures  (signposts)  of  two 
general  kinds.  The  first  focused  on  the  progress  of  the  partnership  in 
achieving  its  innovation  goals  (which  could  indicate  whether  part¬ 
nership  activities  were  leading  toward  results)  and  asked  questions 
such  as: 

•  Have  measurable  outputs  and  outcomes  been  established  to  as¬ 
sess  success  or  failure?  Is  there  an  internal  evaluation  program 
that  will  track  how  close  the  partnership  is  to  meeting  its  goals? 
Do  they  suggest  that  the  partnership  is  meeting  expectations 
along  the  way? 

•  Progress  on  the  technical  core — is  it  succeeding? 

•  How  much  time  did  it  take  to  reach  specific  technological  goals? 
What  does  it  suggest  about  the  ability  to  expedite  the  process  of 
innovation? 

The  second  set  of  measures  was  focused  on  the  characteristics  of  the 
partnership  that  could  indicate  whether  the  partnership  was  devel¬ 
oping  and  using  the  necessary  instruments  in  pursuit  of  the  correct 
actions.  As  many  of  these  are  closely  related  to  the  character  of  the 
partnership,  these  will  be  discussed  in  greater  detail  in  Chapter  Four. 

FROM  INNOVATION  TO  WEALTH 

Most  of  the  workshop  discussions  were  focused  on  the  second  link  in 
the  innovation  chain — the  steps  involved  in  bringing  an  innovative 
application  into  the  commercial  world.  Two  breakout  sessions  de¬ 
voted  considerable  attention  to  the  characteristics  of  commercially 


9In  this  case,  one  needs  to  compare  observed  outcomes  with  those  from  a  counterfac¬ 
tual  world  in  which  the  PFI  program  did  not  support  any  of  the  partnerships;  some 
might  have  found  funding  for  their  partnership,  in  which  case  some  level  of  benefits 
might  still  have  resulted  even  without  the  PFI  program. 
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useful  research,  the  principal  challenges  and  barriers  to  commercial¬ 
ization,  and  the  means  for  eliminating  these  barriers,  discussed  next. 

Characteristics  of  Commercially  Useful  Research 

There  was  general  agreement  on  a  number  of  key  points  regarding 
the  characteristics  of  commercially  useful  research:10 

•  Uniqueness.  Rather  than  thinking  in  terms  of  “research”  or 
“development,”11  commercially  useful  research  was  envisaged  in 
terms  of  creative  ideas  that  could  lead  to  a  new  product  or 
process,  increased  performance  or  cost-effectiveness  for  an 
existing  one,  or  some  other  improvement.  Key  characteristics  of 
innovative  research  included  unique  features  or  a  proprietary 
advantage  over  the  present  state  of  the  art. 

•  Industry- Driven.  Participants  saw  commercially  useful  research 
as  being  industry- driven  in  the  sense  of  meeting  the  needs  of 
companies  working  on  a  specific  area  of  technology  application 
by  providing  research  that  can  improve  a  product  or  process. 

•  Market-Driven.  There  was  general  agreement  that  a  key  charac¬ 
teristic  of  commercially  useful  research  was  that  it  needed  to 
meet  a  real  need  and  offer  a  sufficiently  high  payoff  or  return  on 
investment  in  terms  of  commercial  benefit  to  justify  the 
potentially  high  risk.  Whether  the  research  yielded  incremental 
or  discontinuous  change  or  was  directed  at  a  narrow  or  niche 
market  or  a  broad  industry  or  regional  market,  it  required  an 
identified  end  user  and  customer  and  market  receptivity.  This  in 
turn  necessitated  both  a  tangible  benefit  (e.g.,  a  new  capability, 
increased  performance,  cost  reduction)  and  a  capacity  to  capture 
the  payoff  rapidly  enough  to  meet  company  or  industry  time 
frames.  Participants  argued  that  if  a  commercial  entity  was 
willing  to  provide  resources,  the  research  was  by  definition 


10For  additional  reading  on  when  partnerships  can  be  valuable,  see  Roberts  and  Berry 
(1985).  For  additional  reading  on  criteria  for  building  a  successful  business,  see 
O’Brien  and  Fadem  (1999). 

11  This  was  captured  in  a  question  asked  by  one  of  the  breakout  sessions:  “Is  it 
‘Research,’  ‘Development,’  or  ‘Innovation’?” 
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commercially  viable  but,  over  the  long  term,  a  market  was 
needed  to  support  the  research.12 

Principal  Challenges  or  Barriers  to  Commercialization 

A  number  of  challenges  or  barriers  were  identified  throughout  the 
commercialization  process.13 

First,  simply  identifying  strategic  partners  represented  a  potential 
challenge  for  both  university  and  industry  actors.  This  seemed  to  be 
tied  to  the  belief  that  university  researchers  frequently  did  not  know 
which  firms  might  benefit  most  from  application  of  their  research. 
On  the  industry  side,  many  firms  did  not  fully  understand  the  worth 
of  research,  how  to  integrate  it  into  company  operations,  or — par¬ 
ticularly  in  the  case  of  smaller  firms — which  researchers  were  work¬ 
ing  in  relevant  technology  areas.14 

Another  potential  challenge  was  ensuring  the  requisite  level  of  com¬ 
patibility  of  objectives,  and  mutual  understanding  of  (and  respect 
for)  differences  in  the  objectives  of  different  partners: 

•  On  the  one  hand,  universities  and  industry  were  seen  to  have 
very  different  institutional  cultures  and  motivations  that  could, 
without  adequate  recognition  and  attention,  disrupt  an  other¬ 
wise  productive  partnership.  For  example,  because  the  planning 
horizons  for  companies  are  typically  shorter  than  those  for  uni¬ 
versity  researchers,  time-to-market  often  can  be  a  problem:  It 
can  be  difficult  to  get  work  done  for  a  company  quickly  in  an 
academic  environment,  since  academic  research  by  definition 


12There  also  were  two  “minority  opinions”  on  this  subject.  First,  some  held  that  uni¬ 
versity  researchers  should  identify  problems  and  solutions  rather  than  seeking  to  find 
technology  that  was  commercially  viable;  in  this  view,  commercialization  simply  was 
not  the  business  of  universities.  In  a  similar  vein,  some  felt  that  commercially  success¬ 
ful  applications  had  little  educational  value  for  students.  Still  another  view  was  that 
university  researchers  would  be  placed  in  an  undesirable  (subordinate)  role  by  being 
too  closely  tied  to  commercialization.  See  Ember  (2000)  and  Wasserman  (2000). 

13The  innovation  process  is  filled  with  risks:  Studies  have  shown  that  it  takes  3,000 
raw  ideas,  out  of  which  over  300  novel  ideas  are  identified,  to  produce  one  significant 
commercial  success.  See  Stevens  and  Burley  (1997). 

14This  recognition  occasioned  a  number  of  ideas  about  how  best  to  perform  the  nec¬ 
essary  "matchmaking,”  discussed  below. 
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takes  longer.  At  the  same  time,  the  university  has  a  longer-term 
focus  with  respect  to  both  research  and  education.  Although 
participants  felt  that  the  business  community  as  a  whole  under¬ 
stands  today  better  than  ever  that  training  the  workforce  is  im¬ 
portant  and  that  good  technological  skills  are  needed,  this  ap¬ 
preciation  may  not  find  expression  in  all  firm  partnerships  with 
universities. 

•  On  the  other  hand,  potential  friction  was  seen  to  arise  because 
universities  and  industry  often  were  seen  to  be  after  precisely  the 
same  thing — e.g.,  financial  returns  from  the  exploitation  of  intel¬ 
lectual  property  (IP)  rights.  Universities  increasingly  have  been 
seeking  to  capture  financial  rewards  from  their  IP,  to  the  point 
where  the  issue  has  become  a  source  of  friction  in  industry- 
university  relationships.  Two  recurring  topics  in  the  workshop 
were  the  need  to  clarify  intellectual  property  issues  to  reduce 
these  sources  of  potential  friction  and  the  need  for  universities  to 
look  beyond  financial  returns  on  their  IP  that  could  yield  impor¬ 
tant  benefits  (e.g.,  entrepreneurial  experience,  training  and 
workforce  development,  or  new  lab  equipment)  by  crafting  cre¬ 
ative  win-win  arrangements,  rather  than  inciting  disagreements 
over  how  to  fairly  divide  royalties. 

•  In  a  similar  vein,  industry  partners  frequently  want  to  keep  pro¬ 
jects  confidential  as  long  as  possible,  whereas  university  partners 
want  to  publish  results  as  quickly  and  widely  as  possible.15 

Access  to  various  types  of  resources  also  was  seen  as  a  potential  bar¬ 
rier: 


•  Money.  The  availability  of  capital  was  seen  as  an  important  po¬ 
tential  barrier  to  commercialization,  although  it  also  was  seen  as 
a  potential  area  of  company  advantage,  since  companies  typi¬ 
cally  had  at  least  some  money  that  they  could  use  to  locate  or 
seed  academic  partners. 

•  Personnel  Participants  recognized  the  challenge  of  developing 
or  attracting  experienced  and  skilled  personnel  in  all  roles  and  at 
all  levels  of  the  innovation  enterprise.  These  included  accom- 


15See  Behrens  and  Gray  (2001). 
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plished  leaders  who  could  construct,  articulate,  promote,  and 
coordinate  execution  of  a  coherent  vision  and  carry  it  through 
the  commercialization  process;  managers  who  had  the  necessary 
business  acumen  and  experience  in  integrating  R&D  and  busi¬ 
ness,  and  bringing  technologies  to  market;  and  the  right  number 
and  mix  of  people  in  the  right  location  to  ensure  successful 
product  or  process  development. 

•  Infrastructure.  In  a  similar  vein,  the  lack  of  infrastructure,  in¬ 
cluding  administration  and  management  support,  was  seen  as  a 
potential  barrier  to  successful  commercialization  of  research. 


Means  for  Eliminating  Barriers 

Of  the  various  challenges  to  commercialization  that  workshop  atten¬ 
dees  identified,  perhaps  the  greatest  amount  of  attention  was  de¬ 
voted  to  improving  the  chances  that  prospective  university  and  in¬ 
dustry  partners  would  be  able  to  find  one  another. 

There  appeared  to  be  numerous  opportunities  to  improve  commu¬ 
nications  between  potential  academic  and  industry  partners  that 
could  facilitate  the  identification  of  potential  matches  and  create 
new  opportunities  for  partnerships.  Among  the  means  that  were  dis¬ 
cussed  were  encouraging  a  variety  of  outreach  activities  that  would 
bring  together  the  academic  and  business  communities,  public  re¬ 
porting  of  research  activities  to  improve  industry  awareness  of  po¬ 
tentially  useful  research,  and  the  development  by  universities  of 
marketing  or  sales  strategies  that  target  industries  with  information 
about  researchers  performing  potentially  useful  research. 

Also  viewed  as  important  were  changes  to  university  and  industry 
reward  systems  and  incentives;  without  the  proper  incentives,  some 
argued,  efforts  to  bridge  the  university-industry  gap  could  be  ex¬ 
pected  to  remain  smaller  than  desired.  Among  the  changes  that  were 
suggested  were  revised  university  policies  on  promotion  decisions 
that  would  give  greater  weight  to  patenting  and  other  innovative  ac¬ 
tivity  and  tax  credits  for  industry  partners  that  could  promote  greater 
efforts  by  industry  to  find  university  partners.  Efforts  to  speed  up  the 
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patent  cycle  and  time-to-market  and  those  aimed  at  improving 
knowledge  management  practices  to  facilitate  the  diffusion  of  practi¬ 
cal  knowledge  also  were  seen  as  important  means  of  eliminating  ex¬ 
isting  barriers  to  commercialization. 

Measurement  Issues 

Outcome  Measures.  The  project-level  outcome  measures  that  re¬ 
lated  to  the  successful  commercialization  of  an  innovative  applica¬ 
tion  focused  predominantly  on  indicators  that  an  innovative  appli¬ 
cation  was  being  used  and  yielding  commercial  rewards.  These  in¬ 
cluded  the  number  of  licenses/licensees,  revenue  from  licenses  and 
royalties,  equity  positions,  initial  public  offerings,  buyouts,  sales,  and 
jobs  created. 

At  the  program  level,  a  number  of  suggested  outcome  measures  ad¬ 
dressed  the  broader  question,  “Has  the  overall  PFI  contributed  to  in¬ 
novation?”  Most  of  these  were  viewed  simply  as  summations  of  the 
project-level  outcome  measures  related  to  innovation  that  were 
identified  earlier,  such  as  the  total  number  of  patents  for  all  projects 
supported  by  the  PFI  program  over  some  period  of  time. 

Signposts.  Signposts  for  this  step  included  meetings  between  re¬ 
searchers  and  industry  partners,  meetings  between  partners  and  po¬ 
tential  customers,  other  expressions  of  commercial  interest  in  the 
innovation,  as  well  as  the  actual  provision  of  startup  funding  or  other 
commitments  by  industry  partners  or  customers. 

FROM  WEALTH  TO  NATIONAL  WELL-BEING 

The  final  link  in  the  commercial  innovation  chain  involves  the 
spread  of  benefits  beyond  the  partnership  and  its  immediate  cus¬ 
tomers  to  the  broader  local,  state,  regional,  and  national  economies. 
This  was  viewed  as  taking  place  as  a  result  of  expansion  of  the  inno¬ 
vative  enterprise  that  is  the  focus  of  the  partnership  and  the  expan¬ 
sion  of  firms  or  industries  that  may  benefit  from  the  innovation.  It 
also  could  happen  by  virtue  of  productivity  improvements  that  result 
from  the  innovation. 
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Considering  Comparative  (and  Competitive)  Advantages 

Workshop  participants  suggested  the  potential  for  localities,  regions, 
states,  or  nations  to  compete  with  one  another,  particularly  in  creat¬ 
ing  new  technology  clusters  that  could  lead  to  high-quality  jobs  and 
other  types  of  economic  development  in  the  same  way  that  com¬ 
parative  and  competitive  advantage  play  in  commercial  markets.16 

•  At  the  regional  level,  it  was  seen  as  important  to  mobilize  the  in¬ 
tellectual  and  economic  resources  resident  in  the  region  to  real¬ 
ize  the  opportunity  in  key  emerging  technology  sectors  to  build  a 
sustainable  regional  economy  and  to  compete  with  other  re¬ 
gional  clusters.17 

•  At  the  state  level,  it  was  noted  that  many  states  have  the  same 
sorts  of  objectives  in  fostering  regional  development  based  upon 
private  sector  investment  in  technology  industries.  These  states 
essentially  compete  with  one  another  for  private  and  other  in¬ 
vestment.  This  sort  of  competition  requires  that  a  partnership 
consider  broader  economic  development  strategies  that  include 
private  sector  entrepreneurs  to  establish  the  comparative  advan¬ 
tages  of  one  state  over  another.  This  issue  also  raised  important 
questions  about  the  roles  of  the  private  and  public  sector,  how 
universities  educate  and  train  the  workforce,  and  how  the  state 
can  provide  some  of  the  infrastructure  that  can  confer  competi¬ 
tive  advantage.18 

•  More  broadly  still,  some  argued  that  it  was  important  to  integrate 
across  regions  and  not  to  rely  solely  on  educational  institutions 
within  a  specific  area  or  state;  developing  partnerships  from 


16On  technology  clusters,  see  Swann,  Prevezer,  and  Stout  (1998)  and  Saxenian  (1996). 
A  reason  for  clustering  is  offered  by  Wolfgang  Keller:  Technology  diffusion  is  severely 
limited  by  distance.  See  Keller  (2001). 

17For  example,  one  partnership  saw  Research  Triangle  Park  and  the  Richmond  and 
Northern  Virginia  areas  as  potential  regional  competitors. 

18Analyses  of  state-based  science  and  technology  parks  suggest  that  this  might  not 
always  be  the  case.  An  analysis  of  Research  Triangle  Park  and  University  of  Utah  Re¬ 
search  Park  showed  that  the  former  aimed  to  attract  talent  and  investment  mainly 
from  out  of  state  and  the  latter  focused  on  providing  incentives  to  faculty,  students, 
and  state  residents  to  develop  high-technology  businesses  in  the  state,  with  both  find¬ 
ing  success.  See  Luger  and  Goldstein  (1991). 
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within  the  network  of  former  students  and  a  broader  network 
abroad  was  suggested  as  one  way  of  doing  this.  Some  even  ar¬ 
gued  that  networking  enabled  competitors  to  become  collabora¬ 
tors,  in  which  case  geography  would  not  matter  as  much. 

Measurement  Issues 

Outcome  Measures.  Some  attendees  argued  that  project-level  out¬ 
come  measures  should  include  micro -level  economic  outcomes  such 
as  partnership-related  jobs  created,  although  again  the  distinction 
between  the  micro-  and  macro -levels  was  not  always  clear,  and 
participants  recognized  that  there  are  inherent  difficulties  in  their  es¬ 
timation. 

Whether  the  level  of  analysis  is  an  individual  partnership,  a  network, 
a  locality,  a  state,  or  a  region,  measuring  economic  effects  relies  on 
analyses  that  compare  observed  outcomes  (e.g.,  jobs  created)  with 
some  estimate  of  the  baseline  level  of  economic  performance  that 
would  have  obtained  in  a  counterfactual  world  where  the  PFI  project 
did  not  exist.  In  this  world,  the  partners  might  (or  might  not)  still 
have  met  with  some  level  of  commercial  success,  with  some  level  of 
broader  economic  benefit  resulting.  Ascertaining  the  “value  added” 
from  a  project  is  therefore  quite  difficult,  since  success  is  a  nonnor¬ 
mal  random  variable  and  because  of  the  complexity  and  looming 
uncertainties  in  counterfactual  analyses.19 

Thus,  if  the  assessment  of  micro -level  economic  effects  was  viewed 
as  a  difficult  proposition,  participants  viewed  as  tough  to  impossible 
the  task  of  reliably  assessing  macro-level  economic  effects.  For  the 
most  part,  this  was  because  of  the  same  sorts  of  problems  that 
plagued  the  micro-level  analyses  of  economic  effect:  understanding 
outcomes  that  derived  from  complex  economic  processes  both  in  the 
observed  world  and  in  a  counterfactual  one.  In  a  broader  (state  or 
regional)  setting,  the  number  of  potential  confounding  factors — and 
the  uncertainties — grew.  Of  course,  to  the  extent  that  the  range  of 
effects  could  be  bounded,  they  could  be  treated  as  summative  and 


19For  example,  see  BankBoston’s  effort  to  relate  “MIT-related"  companies  to  jobs  and 
sales.  BankBoston  Economics  Department  (1997). 


24  Building  a  New  Foundation  for  Innovation:  Results  of  a  Workshop 


aggregated  to  estimate  the  range  of  potential  economic  effects  for  the 
PFI  program  as  a  whole.20 

Signposts.  No  distinctions  were  made  between  the  outcome  mea¬ 
sures  and  the  principal  signposts  for  gauging  broader  economic  ef¬ 
fects. 

CONCLUSIONS 

This  chapter  has  provided  a  synthesis  of  workshop  discussions  re¬ 
garding  the  various  phases  of  innovation.  It  suggested,  first,  that  in¬ 
novation  appeared  to  be  most  likely  at  the  seams  between  dis¬ 
ciplines,  technologies,  and  institutions,  with  strong  implications  for 
the  characteristics  of  the  sorts  of  partnerships  that  should  be  nur¬ 
tured,  a  subject  that  is  taken  up  in  greater  detail  in  the  next  chapter. 
It  also  suggested  that  commercialization  was  the  essential  process  by 
which  innovations  can  lead  to  wealth  and  that  the  commercial  con¬ 
siderations  are  the  preeminent  ones  determining  the  stream  of 
benefits  that  may  flow  from  an  innovation.  It  also  suggested  the  pro¬ 
cess  by  which  commercial  innovations  can  have  broader  economic 
effects,  on  productivity,  for  example,  or  job  growth,  although  the  di¬ 
rect  connections  can  be  exceedingly  difficult  to  establish.  The  next 
chapter  shifts  the  focus  to  workshop  discussions  regarding  partner¬ 
ships. 


20See,  for  example,  Salter  and  Martin  (2001),  which  concludes  that  no  simple  model  of 
the  economic  benefits  from  basic  research  is  possible.  For  additional  reading,  see 
Griliches  (1991, 1995)  and  Mansfield  (1991a,  1991b).  Both  have  attempted  to  develop 
estimates  of  return  from  basic  research  to  society. 


Chapter  Four 

CATALYZING  PARTNERSHIPS  TO  ENABLE 
INFRASTRUCTURE  AND  BROADEN 
PARTICIPATION 


When  we  first  set  about  designing  the  PFI  program,  we  had  many 
hours  of  lively  debate  about  how  it  should  be  structured.  What  guide¬ 
lines  should  we  develop  for  prospective  grantees?  What  parameters 
should  we  require  of  the  partnerships?  We  all  knew  that  PFI  would 
best  serve  its  purpose  if  it  attracted  a  wide  variety  of  institutions, 
partnerships,  and  proposed  experiments  and  innovation. 

In  the  end,  we  decided  to  place  very  few  restrictions  in  the  proposal 
description.  We  wanted  to  provide  the  maximum  freedom  possible 
for  grantees  to  be  innovative  about  innovation. . . .  We've  reached 
out  to  find  the  capable  people  with  the  best  ideas  to  begin  the  ex¬ 
traordinary  process  of  transforming  our  innovation  system  to  meet 

the  needs  of  the  21st  Century. 

Joseph  Bordogna,  Deputy  Director,  National  Science  Foundation 

To  paraphrase  one  breakout  session’s  plenary  report:  “The  PFI  aims 
to  catalyze  innovation,  and  partnerships  are  how  we  go  about  doing 
that.”  This  chapter  accordingly  describes  workshop  discussions  re¬ 
garding  three  further  aims  of  the  PFI  program  that  are  related  to 
partnerships  and  their  growth:  catalyzing  partnerships,  catalyzing 
enabling  infrastructure,  and  broadening  participation  in  the  national 
innovation  enterprise. 


25 


26  Building  a  New  Foundation  for  Innovation:  Results  of  a  Workshop 


CATALYZING  PARTNERSHIPS  FOR  INNOVATION 

More  often  than  not,  workshop  participants  described  genuine  part¬ 
nerships  as  dynamic  and  growing  relationships  based  upon  shared 
interests,  trust,  and  an  evolving  technical  relationship.1  Participants 
described  these  relationships  as  necessarily  multifaceted,  including 
senior  researchers,  students,  business  people,  and  others  who  could 
find  common  ground  and  purpose,  and  as  long  term  in  nature. 

Attendees  identified  a  number  of  characteristics  that  were  associated 
with  successful  partnerships  that  can  be  thought  of  in  terms  of  dis¬ 
crete  steps.  These  steps  incorporate  both  broad  issues  of  the  part¬ 
nership's  ends  and  narrower  ones  of  means,  ultimately  leading  to  a 
fuller  understanding  of  the  essential  interdependency  of  all  part¬ 
ners.2  Although  the  following  discussion  suggests  a  top-down — and 
sequential — process,  there  is  obvious  concurrency  and  interdepen¬ 
dency  among  the  steps. 

Establishing  a  Vision 

First,  successful  partnerships  have  a  vision  that  provides  an  image  of 
what  constitutes  success  in  fairly  concrete  terms  (e.g.,  an  incremental 
innovation  that  can  improve  a  product  or  process  or  a  radical  in¬ 
novation  that  can  enable  a  new  one).3  This  can  provide  a  basis  for 
further  specifying  goals  and  objectives,  for  establishing  a  roadmap 
for  realizing  the  vision,  and  for  providing  incentives  to  agents  of 
change. 

Workshop  attendees  placed  a  great  deal  of  emphasis  on  the  devel¬ 
opment  of  a  vision  for  each  partnership,  one  that  in  many  cases 
would  be  the  work  of  a  single  champion  who  could  provide  the  lead- 


1  Participants  in  one  breakout  session  contrasted  a  partnership  in  which  the  parties 
want  the  partnership  to  grow  and  sustain  itself  with  a  “marriage  of  convenience”  in 
which  one  party  just  wants  something  (e.g.,  money).  Mowery  (1998)  describes  key  el¬ 
ements  of  collaborations  and  proposes  features  that  are  associated  with  better  per¬ 
formance. 

2Some  participants  even  believed  that  there  was  a  cyclical  component  in  the  partner¬ 
ship  process:  As  initial  goals  were  achieved,  members  of  the  partnership  would  estab¬ 
lish  new  ones. 

3As  E.  Roger  Novak  of  Venture  Capital  for  Partnerships  put  it  in  his  talk:  “Ideas  are 
plentiful,  visions  are  powerful.” 
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ership  and  overall  coordination  necessary  to  ensure  that  the  partner¬ 
ship  was  a  success.  This  vision  would  need  to  set  a  tone  that  em¬ 
braced  the  most  important  aspirations  of  all  of  the  partners,  while 
providing  a  coherent  statement  of  the  overall  end  state  to  which  the 
partnership  itself  aspired. 

Establishing  Performance  Goals  and  Benchmarks 

There  was  broad  agreement  that  the  partnership’s  vision  needed  to 
be  supplemented  by  a  definition  of  measurable  outcomes  and  ben¬ 
efits  of  the  partnership  that  could  be  used  to  establish  benchmarks 
and  to  measure  progress.4 

Clear,  achievable  measures  are  needed  to  assess  goal  performance 
(e.g.,  results  and  return  on  investment)  and  these  should  include 
measures  that  speak  to  the  quality  of  relationships  in  the  partner¬ 
ship.  The  metrics  should  point  to  the  desired  end-state  for  the  part¬ 
nership,  which  some  argued  should  provide  the  basis  for  an  “exit 
strategy”  that  plans  for  the  obsolescence  of  the  current  project  and  a 
path  to  the  next  one.5 

Identifying  Leaders 

To  realize  the  partnership’s  vision  and  to  make  it  operational,  a  vi¬ 
sionary  and  passionate  leader  is  needed.  This  leader  must  either  be 
brought  in  or  cultivated  from  within  the  partnership.6  In  either  case, 
because  passionate  leaders  are  rare,  expensive,  and  in  high  demand, 
money  must  be  available  to  pay  whatever  the  market  will  bear  for 
their  services.7  The  frequency  with  which  the  topic  of  leadership 
arose  suggested  that  many  participants  doubted  whether  a  partner- 


4We  will  return  to  the  question  of  performance  measures  in  the  next  chapter. 

5This  could  involve  either  a  focus  on  a  new  area  or  incremental  improvements  (line 
extensions)  to  an  existing  line. 

According  to  some,  the  self-interest  of  all  parties  was  important  in  both  the  short  and 
long  term,  whereas  champions  (organizations  and  people)  were  needed  for  the  long 
term. 

7Many  seemed  to  argue  that  the  leader  generally  would  precede  and  lead  the 
development  of  a  partnership,  but  the  possibility  of  a  leader  emerging  from  within  the 
partnership  also  was  recognized. 
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ship  could  even  survive  without  one  or  more  vibrant  leaders  who 
could  champion  the  partnership's  cause. 

A  recurring  theme  in  the  discussions  related  to  the  broadening  of 
discrete  partnerships  to  larger  networks  and  clusters  was  the  need 
for  strong  local  leaders  from  relevant  professional  communities  (e.g., 
university,  industry,  venture  capital,  the  law).  These  leaders  had  to 
be  both  influential  and  powerfully  committed  to  the  region,  capable 
of  knitting  together  all  of  the  elements  that  were  necessary  to  ensure 
a  high  probability  of  success:  strategic  and  business  plans,  private 
capital,  broader  institutional  and  community  support,  and  other  el¬ 
ements.  They  also  needed  to  be  capable  of  building  and  managing 
the  partnership  and  selling  the  partnership  to  the  outside  world  (e.g., 
potential  investors,  the  community,  the  state). 


Recruiting  Committed  Partners 

As  suggested  above,  the  core  of  a  successful  partnership  was  con¬ 
ceived  of  as  comprising  the  minimal  set  of  partners  needed  to  pro¬ 
vide  all  of  the  competencies  and  resources — an  innovative  idea, 
leadership,  one  or  more  technological  approaches  to  its  realization, 
entrepreneurial  and  marketing  skills,  administrative  and  financial 
support,  and  so  on — to  bring  an  innovation  to  market. 

The  overarching  aim  in  finding  partners  was  to  identify  other  parties 
with  common  strategic  interests  and  complementary  strengths  and 
weaknesses: 

•  For  universities,  this  often  meant  looking  for  industry  partners 
with  equipment,  infrastructure,  or  other  needed  resources — not 
necessarily  financial  ones8 — who  might  be  able  to  implement 
research. 

•  For  industry,  the  aim  was  finding  academic  partners  who  had 
relevant  know-how  that  could  make  a  contribution  to  a  joint 
venture. 


8Indeed,  some  participants  felt  that  academics  who  were  going  after  money  were  go¬ 
ing  down  the  wrong  path. 
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For  purposes  of  building  the  partnership,  this  necessitates  a  three- 
step  process  for  building  shared  commitment  based  upon  mutual 
understanding  and  appreciation: 

•  Mutual  Understanding.  Successful  partnerships  were  character¬ 
ized  by  candor  and  mutual  understanding  of  individual  partners’ 
principal  reasons  for  involvement  in  the  partnership,  including 
their  motives,  goals,  and  needs;  the  compatibility  (or  potential 
incompatibility)  of  partners'  interests;  comparative  strengths 
and  weaknesses,  resources,  and  constraints;  and  roles  and  re¬ 
sponsibilities  of  each  member  of  the  partnership.9 

•  Mutual  Appreciation  Based  on  Interdependency .  The  mutual  un¬ 
derstanding  found  in  genuine  partnerships  and  the  alignment  of 
roles  and  responsibilities  based  upon  an  understanding  of  com¬ 
parative  strengths  and  weaknesses  of  the  partners  led  to  an  in¬ 
terdependency  that  was  viewed  as  healthy  and  essential  for  the 
partnership.  If  partners  did  not  really  need  one  another  to  ac¬ 
complish  the  partnership's  goals,  some  averred,  then  they  should 
go  elsewhere.  A  partnership  meant  that  a  partner  really  could 
not  go  anywhere  else  to  accomplish  his  goals. 

•  Shared  Commitment.  In  successful  partnerships,  it  was  critical 
that  the  interdependency  and  essential  need  for  the  capabilities 
of  other  partners  just  described  be  accompanied  by  shared 
commitment — all  partners  need  to  be  highly  motivated  and 
deeply  involved  in  the  partnership.10 

A  network  of  committed  customers  and  other  stakeholders  and  sup¬ 
porters  also  must  be  developed,  providing  positive  feedback  and 
support  for  joint  ventures.  Customers  and  other  stakeholders  and 
supporters  at  all  levels  must  be  continually  engaged  and  apprised  of 
the  status  of  progress. 

University  Strategies  for  Finding  Partners.  A  number  of  elements  of 
potential  university  strategies  for  finding  partners  were  identified: 


9A  prominent  subject  was  ensuring  a  level  of  candor  among  partners  to  reveal  their 
true  motives  rather  than  claiming  purely  altruistic  ones. 

10Words  such  as  “driven"  were  used  to  describe  the  nature  of  the  partners'  commit¬ 
ment,  and  partnerships  were  said  to  require  "a  sense  of  urgency  in  the  strategic 
sense — something  needs  to  happen.” 
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•  “Partnership  friendly”  university  policies  were  seen  to  make  a  big 
difference  in  attracting  industry  partners,  with  universities  hav¬ 
ing  such  policies  able  to  advertise  these  policies  to  attract  indus¬ 
try  partners; 

•  Multidisciplinary  science,  technology,  and  business  centers  and 
university- industry  institutes  were  seen  as  excellent  indicators  of 
a  “partnership  friendly”  environment  that  could  provide  a  po¬ 
tentially  effective  and  efficient  liaison  between  the  university  and 
potential  industry  partners;11 

•  Technology  transfer  offices  can  evangelize  the  university’s  en¬ 
dowments  in  science  and  technology,  including  its  skilled  sci¬ 
ence  and  engineering  workforce,  and  both  senior  researchers 
and  graduate  students; 

•  In  light  of  the  belief  that  universities  need  to  understand  an  in¬ 
dustry  to  target  it,  universities  can  conduct  industry  needs  as¬ 
sessments  and  capacity  assessments  to  ascertain  what  type  of 
technologies  are  needed  and  commercially  viable  and  where 
university  researchers  can  add  the  most  value; 

•  In  relatively  mature  areas  of  technology,  university  researchers 
can  identify  firms  with  the  most  patents  in  an  area;12 

•  Networks  (such  as  alumni  networks)  were  seen  as  a  source  of  po¬ 
tential  industry  partners  who  would  be  sympathetic  to  joint  ven¬ 
tures  with  university  researchers;  such  networks  should  be  en¬ 
couraged  and  used  by  universities; 

•  Showcases  and  industrial  research  fairs  were  viewed  as  poten¬ 
tially  powerful  ways  of  drawing  companies  to  university  events 
and  presenting  opportunities  for  researchers  to  interact  with 
firm  representatives;  such  showcases  could  be  coupled  with 
projects  that  yielded  models  or  pilot  demonstrations  that  had 
potential  for  industry  application. 


Hon  industry-university  collaborations,  see  Industrial  Research  Institute  (1996). 

12This  approach  was  believed  unlikely  to  work  as  well  in  emerging  technology  areas 
and  markets. 
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Industry  Strategies  for  Finding  Partners.  Also  described  were  a 

number  of  potential  industry  strategies  for  finding  academic  part¬ 
ners: 

•  Industry  searches  can  focus  on  universities  with  technology 
transfer  offices,  multidisciplinary  industry- oriented  “centers  of 
excellence/’  universities  with  “partnership  friendly”  policies,  or 
those  that  have  many  patents  or  citations  in  an  area  of  interest; 

•  Concentrating  on  identifying  multidisciplinary  centers  of  excel¬ 
lence  or  university-industry  research  centers  that  focus  on  the 
science,  engineering,  and  technologies  underlying  a  specific  in¬ 
dustry  appeared  to  be  a  more  profitable  approach  than  focusing 
at  the  university  level;  similarly,  focusing  on  schools  with  de¬ 
partment-level  reputation  and  expertise  and  those  that  educated 
particularly  valuable  employees  were  seen  as  potentially  useful 
strategies; 

•  Individual  professors  doing  work  in  relevant  areas  can  be  identi¬ 
fied  by  reputation  or  by  their  publishing  record;  these  academic 
researchers  could  be  engaged  on  specific  technological  problems 
or  challenges  and  invited  to  submit  informal  proposals; 

•  Industrial  fairs  were  seen  as  a  way  of  getting  academics  out  of 
their  departments  and  providing  a  basis  for  evaluating  their 
performance  and  the  prospects  that  they  might  contribute  ma¬ 
terially  to  solving  an  industry  challenge; 

•  Some  firms  simply  show  up  at  a  research  institution  and  see 
what  research  has  been  done  and  what  can  be  used; 

•  Firms  can  use  a  consultant  to  find  the  best  universities  for  work 
in  a  specific  area; 

•  Firms  can  use  employees  who  are  former  graduate  students  as  an 
interface  between  the  business  and  universities; 

•  Representatives  of  firms  can  attend  meetings  of  professional  and 
technical  societies; 
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•  Finally,  the  internet  can  be  used  to  conduct  searches  of  universi¬ 
ties  and  their  expertise.13 

Identifying  Comparative  and  Competitive  Advantages 

Although  there  seemed  to  be  agreement  that  the  broader  market  en¬ 
vironment  narrows  viable  areas  of  research,  participants  felt  that  the 
partnerships  should  aim  to  shape  that  environment.  To  accomplish 
this,  partnerships  need  to  be  guided  by  a  systematic  appraisal  of  their 
own  strengths  and  weaknesses  relative  to  other  commercial  efforts 
that  goes  well  beyond  the  stock-taking  of  individual  partners’ 
strengths  and  weaknesses.  The  partnership  needs  to  identify  its 
comparative  and  competitive  advantages  over  other  technology  de¬ 
velopment  and  industry  efforts,  whether  in  the  larger  context  of  an 
existing  market,  product,  or  process,  or  an  entirely  new  one. 

According  to  workshop  participants,  this  appraisal  should  include  a 
sort  of  market  analysis  that  identifies  customers,  competitors,  prod¬ 
ucts,  processes,  and  underlying  technologies  in  the  area  in  which  the 
partnership  hopes  to  compete  and  identifies  the  unique  characteris¬ 
tics  of  the  partnership’s  efforts  that  will  make  it  competitive.  A  clear 
view  of  the  market  and  potential  competitors  in  that  market  also  was 
seen  as  a  characteristic  of  successful  partnerships.  If  a  partnership 
had  some  things  that  were  unique  and  different  from  its  competitors, 
then  it  could  successfully  compete  and  take  advantage  of  individual 
partners’  expertise.14  The  partnership  also  should  identify  any  re¬ 
maining  elements  (e.g.,  public  relations,  marketing)  that  will  be 
needed  to  bring  the  innovation  to  market,  particularly  if  it  is  an  area 
outside  the  experience  and  expertise  of  existing  partners. 


13For  examples,  see  Yet2.com,  which  acts  as  a  broker  of  technologies,  and  www. 
ninesigma.com,  which  has  developed  a  way  to  find  out  who  is  doing  research  in  what 
areas  and  can  lead  those  interested  in  getting  research  done  for  them  to  proper 
scientists. 

14Workshop  participants  appear  to  have  taken  a  broad  view  of  the  market:  A  market 
could  be  a  particular  locality  or  industiy  niche,  an  entire  industry,  or  a  regional  econ¬ 
omy  that  was  competing  with  other  regional  economies. 
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Resolving  Intellectual  Property  Issues15 

IP  is  the  mechanism  by  which  the  benefits  of  an  innovation  are  ap¬ 
propriated  (owned)  by  one  or  more  parties;  in  general,  because  in¬ 
dustry  is  driven  by  the  profit  motive,  historically  it  has  sought  to  ex¬ 
ploit  intellectual  property  to  a  greater  extent  than  have  universities 
(see  the  box,  below). 

The  disposition  of  intellectual  property — for  purposes  of  workshop 
participants,  primarily  patented  inventions16 — generally  revolved 
around  three  key  questions:  Who  owns  and  is  the  assignee  on  the  IP? 
Who  controls  and  has  the  right  to  commercialize,  sell,  or  otherwise 
use  the  IP?  Who  benefits  through  a  share  in  the  profits  made  from 
the  use  of  the  IP?  In  short,  it  is  the  property  rights  associated  with  the 
intellectual  property,  and  not  the  question  of  who  invented  it,  that  is 
crucial.  In  a  standard  university- industry  collaboration,  three  other 
questions  also  typically  arise:  When  can  the  academic  partner  pub¬ 
lish?  What  level  of  confidentiality  must  be  maintained  in  the  aca¬ 
demic  environment?  And  what  is  the  overhead  rate? 

There  was  broad  agreement  among  workshop  participants  that  the 
benefits  to  researchers  of  intellectual  property  were  not  exclusively — 
nor  even  necessarily  primarily — in  terms  of  financial  sponsorship, 
opportunities  for  consulting,  or  the  revenues  that  might  result  from 
licensing  and  patents.  Indeed,  a  wide  range  of  potential  nonfinancial 
benefits  of  IP  were  cited,  including  internships  for  students  and  early 
recruitment  of  graduate  students  into  industry,  laboratory  equip¬ 
ment  and  other  “in-kind”  benefits,  and  the  potential  treatment  of 
patents  as  “publications”  in  performance  reviews,  promotion,  and 
tenure  decisions.17 


15One  breakout  session  focused  on  the  issue  of  intellectual  property,  and  a  number  of 
others  touched  upon  the  subject  in  their  discussions. 

16Intellectual  property  also  includes  copyrights  and  trademarks,  but  these  were  seen 
as  less  important  for  purposes  of  the  workshop. 

17Many  of  these  benefits  occur  from  partnerships  in  general.  However,  participants 
discussed  at  length  the  benefits  that  were  observed  when  IP  was  also  involved. 
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The  PFI  Program  Seeks  to  Bridge  a  Gap 


Appropriable 

SOURCE:  Hurt  (2001).  Figure  created  by  Jere  Denton  and  used  with  per- 


NOTES:  The  academic  and  nonprofit  sectors  engage  in  activities  the  fruits  of 
which  generally  are  not  appropriated  back  to  them.  They  produce  knowl¬ 
edge  as  a  public  good  in  the  form  of  students,  publications,  and  other  out¬ 
puts  that  can  contribute  to  societal  well-being,  but  they  do  not  generally 
seek_and  often  are  not  able— to  appropriate  benefits  from  these  activities 
for  themselves.  By  contrast,  the  private  sector  engages  in  commercial  activi¬ 
ties,  where  the  benefits  from  intellectual  property  (know-how,  revenues, 
profits,  etc.)  generally  are  appropriable.  PFI  aims  to  fill  a  gap  between  uni¬ 
versity  and  industry  that  can  arise  as  a  result  of  the  university’s  need  for  new 
funding  sources  and  industry’s  need  for  innovative  new  ideas  and  willing¬ 
ness  to  pay  for  them.  It  provides  an  opportunity  for  academic,  nonprofit, 
and  private  sector  actors  to  appropriate  enough  of  the  benefits  to  make  the 
effort  worthwhile  and  sustainable.  For  the  university,  those  benefits  may  be 
in  the  form  of  consulting,  internships  and  on-the-job  training,  funding,  or 
lab  equipment  and  other  infrastructure.  For  the  private  sector,  new  sources 
of  knowledge  can  be  harnessed,  and  skilled  scientists,  engineers,  and  other 
workers  can  be  put  to  work  laying  the  foundations  for  emerging  commercial 
oDDortunities  and  challenges.  _ 
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Although  the  numerous  benefits  seemed  clear  to  participants,  also 
clear  were  the  numerous  issues  that  needed  to  be  resolved  by  each 
member  of  the  partnership  to  ensure  that  IP  did  not  become  a  source 
of  friction.  Participants  noted  that  there  often  were  a  host  of  un¬ 
stated  expectations  in  IP  agreements  (e.g.,  regarding  the  ownership, 
control,  and  benefits  from  IP)  that  equally  often  went  unmet  and  be¬ 
came  a  source  of  friction  between  partners.  The  importance  of 
symmetry  in  the  benefits  also  was  mentioned  as  a  critical  issue,  since 
all  parties  needed  to  have  a  sufficiently  substantial  stake  in  the  out¬ 
come  to  devote  effort  to  achieving  success.  A  lack  of  awareness  re¬ 
garding  what  IP  issues  needed  to  be  worked  out  also  was  deemed  to 
be  an  important  source  of  problems. 

Some  argued  that  the  main  challenge  to  resolving  IP  issues  was  re¬ 
lated  to  the  communication  necessary  to  bridge  gaps  between  part¬ 
ners.  For  example,  cultural  differences  that  arise  from  different  (e.g., 
academic,  industrial,  legal)  backgrounds  and  perspectives  were  seen 
as  important;  universities  need  to  see  themselves  as  partners  rather 
than  owners  of  IP.18  There  also  is  a  lack  of  awareness  of  the  very 
different  capabilities  and  needs  of  small  and  large  businesses;  small 
businesses  were  seen  as  needing  to  own  IP  to  be  attractive  to  others, 
for  their  own  strategic  planning  purposes,  and  to  avoid  licensing  and 
other  costs,  whereas  large  firms  typically  have  law  departments  that 
established  IP  guidelines  and  have  established  ways  of  doing  busi¬ 
ness  with  universities.  The  best  means  for  resolving  disagreements 
was,  not  surprisingly,  fostering  communication  and  mutual  under¬ 
standing  among  all  involved  organizations  and  parties.  Placing  time 
limits  on  the  ownership  of  the  IP  also  was  seen  as  a  means  for  reduc¬ 
ing  long-term  risks  among  the  agreeing  parties.  By  limiting  the  own¬ 
ership,  partners  would  have  the  ability  to  renegotiate,  at  a  fixed 
point,  the  various  equities  for  that  IP  (partners  would  not  be  forever 
signing  away  their  rights). 

Further,  there  frequently  were  tensions  between  the  ownership  of 
the  IP  and  the  benefits  derived  from  the  IP;  IP  generated  at  academic 
institutions  typically  remains  the  property  of  the  institution,  but  the 


18For  example,  proprietary  information,  confidentiality  requirements,  and  reliability 
and  validity  of  data  can  mean  different  things  to  different  people. 
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tax  code  limits  the  ability  of  nonprofit  institutions  funded  with  tax- 
exempt  bonds  to  share  in  the  financial  profits  of  IP.19  Of  course, 
such  restrictions  generally  do  not  apply  to  industry,  but  the  result  is 
nevertheless  that  most  IP  is  either  not  commercialized  at  all  or  is  un¬ 
derutilized,  with  no  one  gaining  benefits  from  it.20 

The  institutional,  cultural,  and  legal  roots  of  these  differences  sug¬ 
gested  that  changes  might  be  needed.  There  was  recognition,  for 
example,  that  in  many  universities  a  cultural  change  would  be 
required  to  change  the  reward  system  so  that  innovative  activities 
were  valued  as  highly  as  standard  performance  measures  such  as 
publications  in  peer-reviewed  journals;21  absent  such  a  cultural 
change,  workshop  participants  were  somewhat  pessimistic  that  in¬ 
novative  activity  could  be  sufficiently  encouraged  and  supported.  In 
a  similar  vein,  it  was  noted  that  a  university's  mission  typically  was  to 
develop  new  knowledge  and  not  to  go  about  identifying  new  cus¬ 
tomers. 

Developing  Plans 

Planning  considerations  play  a  number  of  prominent  roles,  including 
developing  strategic  implementation  plans  that  relate  goals  to  infras¬ 
tructure  and  human  resource  issues;  developing  communications 
strategies;  managing  the  time  of  key  stakeholders  to  ensure  that  their 
time  is  not  wasted;  and  planning  for  the  ultimate  institutionalization 
of  the  partnership  or  network  so  that  its  processes  and  benefits  can 
be  continued. 


19On  the  other  hand,  the  Bayh-Dole  Act  of  1980  (amended  in  1986)  enabled  nonprofit 
institutions  to  receive  patents  on  the  results  of  publicly  funded  research  and  has  been 
credited  with  the  resulting  significant  expansion  in  universities’  efforts  to  support 
patenting,  licensing,  and  transfer  to  industrial  firms  of  university  research  results,  as 
well  as  providing  incentives  for  university-industry  partnerships.  See  Mowery  (1998). 

20There  is  a  growing  trend  for  industry  to  take  IP  not  being  commercialized,  donate  it 
to  a  university  (as  a  tax  write-off),  and  in  some  cases  to  lend  researchers  to  help  the 
university  develop  the  technology  even  further  and  possibly  commercialize  it. 

21A  yearly  luncheon  recognizing  faculty  who  have  patent  inventions  was  proposed  as 
an  example. 
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There  also  seemed  to  be  broad  agreement  that  the  partnership 
should  be  treated  as  a  business,  which  meant  that  the  partnership 
should  have  a  business  model  and  plan.22  According  to  workshop 
participants,  these  should  provide  potential  investors  and  others 
with  an  idea  of  the  potential  value  of  the  innovative  activity,  judged 
on  the  basis  of  the  estimated  size  and  identity  of  the  customer  base 
in  the  relevant  market,  the  nature  of  competition  and  demand  in  the 
market,23  and  analysis  demonstrating  that  the  expected  rewards 
justified  the  expected  risks.  The  plan  also  should  lay  out  the  steps — 
and  costs — involved  in  bringing  the  innovative  product  or  process  to 
market,  including  how  resources  are  to  be  allocated  to  accomplish 
the  partnership’s  goals. 

Competing,  Communicating,  and  Resolving  Conflicts 

When  a  partnership  has  addressed  the  foregoing  issues,  it  is  ready  to 
compete,  but  the  continued  integrity  and  viability  of  the  partnership 
can  be  challenged  as  a  result  of  unforeseen  developments  and  the 
changing  motives  of  the  partners.  This  requires  continued  commu¬ 
nication  among  all  partners  and  conflict  resolution  procedures. 

Competing.  Competing  consists  of  bringing  the  innovative  product 
or  process  to  market  in  competition  with  other  products  or  pro¬ 
cesses. 

Communicating.  Effective  communication  within  the  partnership  is 
necessary,  as  is  promotion  of  success  outside  the  partnership.  Com¬ 
munity/stakeholder  buy-in  that  supports  the  enterprise,  including 
frequent  meetings  with  stakeholders  and  other  means  for  overcom¬ 
ing  fragmentation  and  isolation,  is  needed  to  mobilize  organizations 
and  resources.  Moreover,  the  “selling  of  success” — celebrating  suc¬ 
cess  stories — was  viewed  as  crucial  from  the  outset,  and  identifying 
and  developing  key  audiences  for  marketing  and  public  relations 


22A  typical  conception  of  a  business  model  is  a  one-page  paper  that  describes  how  all 
of  the  elements  of  a  business  work  together  to  build  marketplace  advantage  and  com¬ 
pany  value.  The  business  model  typically  describes  its  intended  customers  and  its 
marketing  strategy  for  reaching  them. 

23The  willingness  of  customers  to  pay  for  the  innovation  was  specifically  identified  as 
being  of  interest. 
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was  viewed  as  an  instrumental  means  for  drawing  in  additional  re¬ 
sources.  Since  many  principal  investigators  (Pis)  lack  expertise  in 
effective  public  relations,  some  argued  that  a  separate  budgeting  cat¬ 
egory  needs  to  be  created  for  public  relations,  and  media  contacts 
need  to  be  included  in  the  network. 

Resolving  Conflicts.  As  discussed  above,  participation  in  a  partner¬ 
ship  connoted  an  essential  compatibility  in  the  interests  of  universi¬ 
ties  and  industry:  The  innovative  capacity  of  universities  could  be 
harnessed  to  the  proven  technology  of  the  private  sector  and  made 
functional,  manufacturable,  and  scaleable.  However,  as  was  de¬ 
scribed  above,  a  number  of  potential  areas  of  incompatibility  were 
seen  to  offer  the  potential  for  university- industry  friction,  including 
differing  time  horizons  and  priorities,  competing  claims  for  royalties 
from  intellectual  property,  and  differences  over  the  timing  and  scope 
for  the  dissemination  of  research  results.  In  recognition  of  these — 
and  other-potential  flash  points,  workshop  participants  advocated 
a  fairly  sensible  approach. 

At  the  outset  of  a  partnership,  all  partners  should  put  their  personal 
agendas  on  the  table  and  ascertain  the  extent  to  which  everyone’s 
needs  can  be  met.  In  some  cases,  areas  of  potential  friction  requiring 
tradeoffs  may  be  creatively  redirected  into  more  productive  ar¬ 
rangements.  Rather  than  competing  over  shares  of  royalties,  for  ex¬ 
ample,  universities  might  seek  laboratory  equipment  or  other  in- 
kind  investments,  access  to  firm  resources,  consulting  opportunities 
for  faculty,  or  employment  opportunities  for  students.  Similarly,  to 
the  extent  that  universities  are  willing  to  modify  performance  and 
promotion  criteria  to  weight  patents  and  other  evidence  of  innova¬ 
tive  research  as  heavily  as  peer-reviewed  publications  are  weighted, 
this  can  eliminate  a  source  of  potential  friction  that  arises  from  the 
“publish  or  perish”  reward  system  faced  by  most  academic  re¬ 
searchers  and  leads  to  a  desire  for  speedy  publication  of  results.  Dis¬ 
putes  that  arise  later  in  the  partnership  simply  should  be  resolved 
with  reference  to  what  was  described  by  one  breakout  session  as  a 
“hierarchy  of  values”  that  weighed  the  issue  in  contention  against  the 
benefits  each  party  received  from  the  partnership,  the  shared  values 
of  the  partnership,  and  the  interdependence  of  the  partners. 
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CATALYZING  ENABLING  INFRASTRUCTURE  TO  FOSTER 
LONG-TERM  INNOVATION 

The  PFI  program  also  aims  to  catalyze  the  creation  of  the  infrastruc¬ 
ture  that  can  sustain  and  nurture  the  spread  of  innovative  activity 
over  the  long  term.  Workshop  deliberations  described  infrastructure 
of  three  general  kinds:  that  related  to  developing  human  capital,  de¬ 
veloping  networks,  and  providing  direct  support  for  the  innovation 
enterprise. 

Developing  Human  Capital 

To  educate  and  train  human  capital  for  the  research  enterprise, 
workshop  participants  saw  the  requirement  for  an  innovative  educa¬ 
tional  process  whose  most  important  goal  was  serving  the  student. 
Such  a  program  should  be  focused  on  creating  an  environment  that 
rewards  innovation  and  is  characterized  by  passion  and  a 
commitment  to  success. 

This  process  would  identify  talent  as  early  as  possible  (from  elemen¬ 
tary  school  on),  create  good  motivational  teachers,  and  embrace  ver¬ 
satility  (e.g.,  affirmative  action  programs,  recognition  of  cultural  dif¬ 
ferences)  to  ensure  that  it  was  drawing  upon  the  largest  possible  pool 
of  new  talent.  As  students  progress,  it  is  critical  to  make  subjects  real 
with  hands-on  education  and  training  and  to  demonstrate  the  con¬ 
nections  between  popular  programs  and  how  they  had  progressed  to 
that  point.  At  the  university  level,  the  most  important  role  of  the  uni¬ 
versity  was  seen  as  educating  and  training  young  scientists  and  engi¬ 
neers;  good  science  and  engineering  students  need  to  be  recruited 
and  then  retained24  and  brought  into  partnerships  with  industry. 
And  internships  should  work  in  both  directions:  faculty  and  student 
internships  in  industry  and  industry  internships  in  universities.25 
Finally,  young  scientists  and  engineers  need  to  serve  as  agents  of 
change,  i.e.,  by  diffusing  know-how  and  innovation. 


24For  a  recent  report  on  the  subject,  see  National  Science  Foundation  (1999). 

25The  NSF’s  Research  Experiences  for  Undergraduates  (REU)  program,  which  pro¬ 
vides  a  framework  for  internships  with  industry,  was  mentioned  as  a  good  model  in 
this  regard,  as  was  the  PFI  program. 
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To  accomplish  this,  workshop  participants  viewed  as  critical  early 
participation  from  all  stakeholders  and  buy-in  from  the  top  for  the 
principle  that  the  most  important  goal  was  serving  students.  And 
buy-in  could  be  quite  literal:  It  could  require  investment  in  addi¬ 
tional  plant  and  teachers  for  traditionally  underserved  populations; 
buy-outs  of  faculty  time  for  sabbaticals,  mentoring,  and  other  activi¬ 
ties;  or  monetary  rewards  for  excellence  in  teaching  or  research. 

The  education  and  workforce  challenges  that  were  identified  ranged 
from  simply  keeping  students  in  the  state  to  paying  for  program  co¬ 
ordinators,  facility  space,  equipment,  etc.,  and  reaching  all  stake¬ 
holders,  especially  minority-owned  businesses.  At  a  macro-level, 
there  also  are  longer-term  workforce  challenges  related  to  ensuring 
that  the  nation  can  develop  (or  attract  and  retain)  needed  science 
and  technology  (S&T)  workers  in  key  areas  (see  the  box,  below).26 

Concern  also  was  expressed  about  the  challenges  arising  from  ad¬ 
verse  demographic  trends,  including  an  increasing  ratio  of  pension¬ 
ers  to  workers  and  chronic  problems  with  the  K-12  educational  sys¬ 
tem.27 

Creating  Networks  that  Embed  Social  Capital 

The  intellectual  capital  and  know-how  embodied  in  young  scientists 
and  engineers,  honed  through  advanced  education  and  training, 
must  be  embedded  in  social  networks  characterized  by  shared 
commitment  and  trust.  Such  networks  can  be  built  only  through  ex¬ 
tended  interactions  and  problem-solving  and  represent  a  form  of 


26Future  workforce  challenges  already  have  been  recognized  and  are  the  subject  of 
high-level  attention.  See  National  Science  and  Technology  Council  (2000).  Addi¬ 
tionally,  the  National  Science  Board  currently  has  a  task  force  that  is  examining 
national  workforce  policies  for  science  and  engineering.  See  National  Science 
Foundation  (2000),  available  at  http://www.nsf.gov/nsb/documents/2000/nwp004/ 
nwp004.htm.  In  considering  the  issue,  however,  one  needs  to  consider  the  capacity  of 
labor  markets  themselves  to  correct  for  increased  demand  by  raising  wages.  See  David 
and  Hall  (2000). 

27See  Good  (2001).  For  an  industry  perspective  on  the  challenges  faced  by  the  K-12 
system  and  some  suggested  remedies,  see  Popper,  Wagner,  and  Larson  (1998),  pp. 
108-109.  For  a  perspective  on  workforce  issues,  see  National  Research  Council  (1998), 
especially  pp.  17-24, 46-47. 
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Ensuring  an  S&T  Workforce  for  Future  Areas  of  Innovation 

According  to  a  presentation  from  Mary  Good3,  U.S.  policymakers  need  to 
look  at  the  implications  of  a  U.S.  S&T  workforce  that  is  both  aging  and 
changing  in  its  composition.  To  illustrate  why  the  United  States  should  be 
concerned,  she  used  the  following  example. 

Many  believe  that  the  next  generation  of  computing  will  be  based  on 
quantum  computing,  and  Hewlett-Packard  (HP),  IBM,  Lucent,  and  many 
universities  accordingly  have  invested  heavily  in  quantum  computing 
research.  Looking  at  members  of  the  Quantum  Science  Research  at  HP's 
laboratories,  however,  two  striking  observations  can  be  made.  First,  all  of 
the  U.S.  members  are  over  the  age  of  45;  and  second,  all  of  the  younger 
ones  are  non-U.S.  citizens.  This  pattern  also  has  been  observed  in  the 
universities  that  are  doing  this  type  of  research. 

As  Good  pointed  out,  this  is  just  one  anecdote,  in  only  one  area  of  S&T,  but 
it  does  raise  several  important  policy- relevant  questions.  How  can  the 
United  States  reverse  adverse  trends  that  may  be  occurring  in  core  areas  of 
S&T  and  better  ensure  that  the  nation  will  have  a  future  workforce  with 
enough  qualified  workers  in  these  areas?  How  can  the  nation  draw  enough 
U.S.  students  into  the  science  and  engineering  workforce  to  ensure  a 
robust  indigenous  capability  in  what  appear  to  be  emerging  core  areas  of 
S&T?  Failing  that,  how  can  the  United  States  ensure  the  availability  of  H-l 
visas  and  immigration- friendly  policies  for  foreign  students  who  are  doing 
research  in  core  S&T  areas  in  U.S.  universities  and  who  might  remain  in  the 
United  States  thereafter?  If  there  are  too  few  American  students,  and 
industry  cannot  hire  foreign  students,  where  will  industry  and  universities 
find  the  expertise  they  need? 

Although  there  are  no  simple  answers  to  these  questions,  they  are  among 
those  that  the  United  States  must  consider  and  take  action  on  to  ensure  the 
necessary  intellectual  capital  to  keep  the  United  States  competitive  in  the 
global  market. 


aGood  (2001). 
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social  capital  that,  in  some  workshop  participants'  minds,  seemed  to 
be  the  most  important  type  of  infrastructure  of  all. 

As  suggested  by  the  earlier  discussion  of  what  is  required  to  catalyze 
innovative  partnerships,  to  create  such  networks  requires  a  substan¬ 
tial  investment  of  time  and  resources  over  an  extended  period  of 
time.  Although  they  may  begin  with  a  specific  project  in  mind,  such 
networks  may  grow  and  expand  beyond  the  terms  of  the  original 
partnership  and  yield  additional  relationships,  projects,  and  partner¬ 
ships. 

The  development  of  this  sort  of  infrastructure  is  enhanced  by  leaders 
and  organizations  that  undertake  the  full  range  of  activities  that  can 
improve  the  prospects  that  researchers  and  business  people  with 
compatible  interests  simply  can  meet  one  another,  and  by  a  track 
record  of  successful  partnerships  that  yield  desired  benefits  to  all 
participants  and  impel  them  to  expand  the  network  into  other  types 
of  partnerships  relevant  to  the  engineering  piece  of  the  enterprise.  It 
also  is  enhanced  by  successful  marketing  and  public  relations  efforts 
that  celebrate  successes,  thereby  drawing  other  parties  into  the  net¬ 
work,  and  in  the  dissemination  of  innovative  practices  that  reflect 
learning  curve  phenomena. 

Providing  Physical  Facilities  and  Support 

A  base  of  operational  support  is  essential,  and  sustainability  cannot 
exist  without  such  operational  support.  The  network  therefore  needs 
to  provide  this.  Ultimately  this  was  viewed  as  a  resource  issue — a  di¬ 
versified  base  of  private  investment  providing  support  for  all  needed 
components.  Among  the  components  identified  were: 

•  An  Incubator.  At  the  most  basic  level,  innovation  was  seen  to  re¬ 
quire  a  place  that  can  facilitate  routine  exchanges  between  the 
business  community  and  the  university,  where  partners  can 
meet  and  work.28  This  can  be  accomplished  through  a  host  of 


280ne  study  found  R&D  geographic  spillover  effects  in  France  and  concluded  that 
"(t)his  trend  towards  a  proximate  localisation  can  be  explained  by  the  complexity  of 
the  R&D  process,  which  reflects  the  need  to  co-ordinate  a  set  of  heterogeneous 
competencies  (combinatory  complexity)  and  the  intensity  of  technological  changes 
(technological  complexity).”  See  Carrincazeaux,  Lung,  and  Rallet  (2001).  On  the  other 
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means,  ranging  from  informal  open  door  policies  that  stem  from 
good  university-industrial  relations  to  formalized  university- 
industry  research  centers.  Whether  formal  or  informal,  such  a 
location  also  can  provide  a  venue  for  meetings,  whether  for  part¬ 
ners  or  extension  efforts  to  educate  and  build  community. 

•  Technical  Support.  Beyond  the  faculty  researchers  and  students 
who  are  doing  the  research  and  their  industry  partners,  a  range 
of  issues  requiring  subject  matter  expertise  are  likely  to  be  be¬ 
yond  the  capabilities  of  the  immediate  partnership.  Supporting 
infrastructure  accordingly  may  include  an  office  that  provides  in- 
house  legal  (e.g.,  patent  assistance  and  IP  policies),  financial, 
marketing,  and  public  relations  expertise  that  can  generate  sup¬ 
port  from  newspapers  and  business  publications. 

•  Management  and  Administrative  Support.  Beyond  the  special¬ 
ized  expertise  just  described,  a  range  of  basic  management  and 
administrative  functions — from  managing  personnel,  payroll, 
and  budgets  to  providing  clerical,  reception,  and  other  services — 
need  to  be  performed  by  competent,  well-trained  staff  to  enable 
the  partners  to  focus  on  research  and  commercialization. 

•  Laboratory  Capacity  and  Other  Physical  Plants .  Needed  infras¬ 
tructure  also  can  include  various  sorts  of  physical  plants,  includ¬ 
ing  computers,  lab  equipment,  machine  tools,  and  laboratory  or 
other  working  space. 

•  Communications.  Infrastructure  also  needs  to  include  support 
for  a  variety  of  alternative  means  of  communication — 
LISTSERVs,  websites,  email  aliases,  moderated  email  discus¬ 
sions,  mailings,  and  other  means — both  among  the  partners  and 
between  partners  and  other  stakeholders.29  This  may  include 
capabilities  to  make  videos  and  CD-ROMs  to  assist  marketing  ef¬ 
forts. 


hand,  Love  and  Roper  provide  a  skeptical  view  of  the  importance  of  location  and 
network  effects  on  the  success  of  innovation  for  manufacturing  plants  in  three  other 
European  countries.  See  Love  and  Roper  (2001). 

29For  example,  participants  mentioned  a  website,  mainscience.org,  which  includes 
proposals  and  abstracts  on  research  from  researchers  and  reaches  3,000  people  a 
week. 
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•  Reliable  Sources  of  Capital.  Finally,  needed  infrastructure  in¬ 
cludes  venture  capital  firms,  angel  investors,  commercial  banks, 
government,  foundations,  and  other  sources  (e.g.,  Small  Busi¬ 
ness  Innovative  Research  (SBIR)  grants)  that  can  provide  start¬ 
up,  bridge,  or  other  funding. 

Beyond  the  fundamental  elements  of  needed  infrastructure  just  de¬ 
scribed,  some  argued  that  the  PFI  also  needs  to  attend  to  a  core 
constituency — members  of  Congress.  Given  that  the  federal  gov¬ 
ernment's  role  in  promoting  innovation  is  a  politically  controversial 
topic  on  Capitol  Hill,  it  is  necessary  to  ensure  that  elected  leaders 
understand  and  support  the  PFI  program’s  aims,  that  the  program 
address  elected  leaders'  potential  concerns,  and  that  there  are  no 
disconnects  between  project  and  program  metrics  and  the  political 
view  of  the  program’s  goals.  According  to  some,  a  tight  linkage 
between  project  and  program  goals  and  metrics  and  a  broader 
communication  strategy  were  needed  for  the  PFI.  This  strategy 
should  include  conversations  with  congressional  representatives 
regarding  what  they  expect  out  of  this  program  and  what  metrics  are 
appropriate  (or  inappropriate)  for  evaluating  the  program. 

Challenges  and  Barriers  to  Sustainability 

Workshop  participants  identified  a  number  of  key  challenges  or  bar¬ 
riers  to  the  emergence  of  these  characteristics.  These  include  turf 
battles  among  stakeholders;  burn  out;  clashes  with  university  culture 
reflected  in  institutional  reward  systems  and  rigid  policies  regarding 
intellectual  property,  startups,  private  sector  engagement,  and  other 
matters;  maintaining  an  ability  to  learn  from  other  participants;  hav¬ 
ing  the  right  people  in  the  right  place  at  the  right  time;  and  the  va¬ 
garies  of  politics. 

Measurement  Issues 

Outcome  Measures:  Capacity.  Another  difficult-to-measure  out¬ 
come  was  the  marginal  increase  in  aggregate  capacity  that  had  re¬ 
sulted  from  the  PFI  program.  At  the  project  level,  this  was  discussed 
in  three  different  forms.  First,  increased  capacity  was  discussed  in 
terms  of  the  actual  infrastructure  (laboratory  equipment,  facilities, 
etc.)  that  had  been  added  as  a  result  of  the  partnership.  Second,  it 
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was  discussed  in  terms  of  the  value  added  to  individual  partners'  in¬ 
frastructure  that  arose  from  exploiting  the  partnership's  synergies. 
Third  was  training  and  workforce  development,  discussed  largely  in 
terms  of  providing  research  professors  with  opportunities  to  apply 
know-how  to  commercial  problems  and  opportunities  for  graduate 
students  to  develop  know-how  regarding  technology  application  and 
entrepreneurship. 

Capacity  that  had  been  built  as  a  result  of  the  PFI  program — whether 
in  terms  of  infrastructure  or  of  training  and  workforce  develop¬ 
ment — was  seen  as  a  summative  outcome  at  the  program  level;  to 
the  extent  that  the  measurement  problems  could  be  resolved  at  the 
project  level,  however,  it  generally  appeared  to  be  a  simple  matter  of 
aggregation  for  the  program  as  a  whole. 

Outcome  Measures:  Sustainability.  Although  their  measurement 
was  not  discussed  in  much  detail,  a  number  of  suggested  outcomes 
were  identified  as  being  of  interest  for  assessing  the  sustainability  of 
the  partnerships.  These  included  outcomes  related  to  the  survival 
and  self-perpetuation  of  the  partnership;  project  staff  in  the  com¬ 
munity;  success  at  fund-raising  activities;  cross-membership  on 
boards  and  other  organizations;  the  growth  of  the  partnership  (i.e., 
its  ability  to  attract  new  people  and  organizations  to  join  the  partner¬ 
ship);  partnering  for  other  grants  and  continuation  even  after  a  lack 
of  success  in  receiving  these  grants;  and  the  emergence  of  new  inno¬ 
vation-oriented  partnerships. 

For  a  host  of  reasons,  assessing  the  sustainability  of  each  partnership 
was  seen  as  a  challenging  enterprise.  First,  there  was  some  concern 
that  the  short  (two  to  three  year)  time  horizon  for  PFI  grants  proba¬ 
bly  was  too  short  a  window  for  assessing  sustainability;  the  sustain¬ 
ability  of  a  partnership  might  not  be  known  until  five  or  even  10 
years  after  its  creation.  Accordingly,  workshop  participants  wanted 
to  know  how  those  involved  in  the  partnership  turned  out  five  years 
later — whether  they  stayed  in  the  game  and  pursued  this  sort  of  ac¬ 
tivity  well  after  the  grant  ended. 

Another  reason  for  the  difficulty  was  that  a  focus  on  the  sustainabil¬ 
ity  of  the  partnership  might  obscure  the  importance  of  the  larger  goal 
of  catalyzing  innovation;  in  this  view,  it  was  the  quality  of  the  inno¬ 
vation  that  mattered,  not  whether  a  partnership  survived  at  some 
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nominal  level.30  This  point  recognized  the  transitivity  and  “creative 
destruction”  that  are  inherent  in  the  innovation  enterprise:  A  part¬ 
nership  that  makes  sense  for  today’s  innovative  project  may  not 
make  sense  for  tomorrow's.  But  it  also  pointed  to  the  potential  im¬ 
portance  of  partnerships  evolving  into  broader  networks  that  might 
link  compatible  partners  who  can  choose  to  combine  and  recombine 
in  different  ways.31  This  latter  view  of  partnerships — as  seeds  that 
have  the  potential  for  evolving  and  growing  into  a  broader  network 
that  could  itself  catalyze  additional  partnerships  and  innovative  ac¬ 
tivity — was  one  that  seemed  to  be  widely,  if  only  tacitly,  embraced  by 
workshop  participants,  and  one  that  was  not  explored  in  much  detail 
in  the  workshop.32 

Sustainability  of  the  partnerships  across  the  entire  PFI  program  also 
was  a  summative  outcome  measure:  To  the  extent  that  the  difficul¬ 
ties  in  measuring  the  sustainability  of  individual  projects  could  be 
resolved  at  the  project  level,  it  was  a  simple  matter  of  indicating  the 
number  or  proportion  of  projects  that  were  judged  as  sustainable. 

Signposts.  Also  suggested  were  measures  that  addressed  a  number 
of  issues  related  to  the  partnerships  themselves,  including: 

•  Organizational  Issues.  How  was  the  partnership  formed?  Is  the 
leadership  team  in  place?  Is  the  leadership  structure  of  the  part¬ 
nership  settled?  Can  it  be  easily  described  to  others  so  that  they 
understand  how  the  partnership  is  organized  and  how  it  works? 
Are  the  necessary  administrative  arrangements  in  place  (e.g., 
budget,  IP,  support  staff)?  Are  information  infrastructure,  per¬ 
sonnel,  other  resources  “integrated”  in  the  operation  of  the  part¬ 
nership  (integration  needs  to  be  measured)?  Are  policies  and 
rules  sufficiently  flexible? 


30One  breakout  session  asked  how  one  compared  a  partnership  that  survived  but  did 
not  succeed  at  generating  an  innovation  with  one  that  was  successful  in  innovating 
but  where  the  partnership  dissolved. 

31On  innovation  networks,  see  Freeman  (1991). 

32The  evaluation  of  sustainability  would  need  to  take  into  account  the  intended  life  of 
the  partnership,  whether  it  is  a  short-term  project,  which,  when  its  objective  is 
achieved,  is  dissolved  or  whether  it  a  longer-term  relationship  that  is  being 
established.  Evaluating  sustainability  should  be  based  on  when  the  goals  are  to  be 
achieved. 


Catalyzing  Partnerships  to  Enable  Infrastructure  and  Broaden  Participation  47 


•  Agreements.  Have  the  necessary  agreements  on  benefits, 
burden-sharing,  and  other  issues  (e.g.,  regarding  intellectual 
property)  been  negotiated?  What  have  members  of  the  partner¬ 
ship  agreed  to  share?  Dollars?  Risk?  Resources?  Graduate  stu¬ 
dents?  Professors?  Measures  are  needed  to  look  at  the  quality  of 
relationship.  How  long  did  it  take  to  come  to  an  agreement? 

•  Finances.  Have  start-up  funding  needs  been  identified?  Have 
sources  of  support  been  identified,  including  internal  resources 
(including,  e.g.,  “skunk  works’'),  grant  funding  (federal,  state,  lo¬ 
cal),  philanthropy,  private  sector  (angels,  venture  capital,  corpo¬ 
rate)?  Have  sales  and  other  revenue  sources  been  identified?  Are 
budgets  and  finances  transparent  and  able  to  be  tracked?  Is 
there  flexibility  in  moving  funds?  Have  processes  been  estab¬ 
lished  for  managing  in-kind  and  matching  funding  and  for  link¬ 
ing  in-kind  support  to  hard  cash? 

•  Plans.  Is  there  a  project  plan?  Is  there  a  business  plan  that  as¬ 
sesses  the  market  and  the  demand  for  the  innovation?  Have  po¬ 
tential  testing/ beta  sites  been  identified  in  the  business  plan?  Is 
there  a  financial  plan?  Is  there  agreement  regarding  the  pro¬ 
cesses  for  modification  of  these  plans?  Do  the  plans  reflect  rea¬ 
sonable  expectations?  Is  there  a  risk  analysis  scheme? 

•  Communications.  Are  the  partners  actively  engaged?  Does  the 
partnership  hold  regular  meetings?  Is  there  a  process  for  com¬ 
munication?  Are  partners  communicating  well?  Are  there  provi¬ 
sions  for  talking  to  top  management  to  resolve  problems?  Have 
measures  for  outreach  activities  been  established?  Does  the 
partnership  produce  products  for  dissemination? 

•  Diversity.  What  level  of  participation  is  there  from  traditionally 
underserved  populations  and  institutions? 

Workshop  participants  seemed  somewhat  divided  on  these  mea¬ 
sures,  however.  On  the  one  hand,  there  was  an  interest  in  under¬ 
standing  the  partnerships  themselves  in  ways  that  could  generate  the 
information  that  can  help  participants  in  their  activities  and  help  the 
NSF  in  further  refining  the  PFI  program.  On  the  other  hand,  there 
also  was  a  clear  recognition  that  successfully  encouraging  partici¬ 
pants  to  keep  good  records  cannot  only  be  difficult  but  it  can  divert 
their  focus  from  innovative  activities  to  administrative  and  docu- 
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mentary  ones.  The  solution  for  some  was  to  be  found  not  in  large 
and  complex  survey  instruments  but  in  identifying  a  few  “nuggets” 
that  needed  to  be  reported  to  track  the  progress  of  the  partnerships, 
for  identifying  best  practices,  etc. 

BROADENING  PARTICIPATION  IN  THE  NATIONAL 
INNOVATION  ENTERPRISE 

A  final  aim  of  the  PFI  program  is  to  broaden  the  participation  of  un¬ 
derrepresented,  disadvantaged,  or  underserved  individuals  and  insti¬ 
tutions.  Workshop  participants  discussed  this  issue  almost  entirely 
in  terms  of  specific  means  for  measuring  project-  and  program- level 
success  at  promoting  this  aim.  Accordingly,  we  turn  directly  to  the 
issue  of  measurement. 

Measurement  Issues 

Outcome  Measures.  At  the  level  of  the  individual,  the  PFI  program 
aims  to  expand  opportunities  for  students  and  professors  who  are 
women,  minorities,  and  members  of  other  demographic  groups;  at 
the  level  of  the  community  and  region,  it  aims  to  encourage  new 
bases  for  economic  and  workforce  development  in  less  well-en¬ 
dowed  locales;  and  at  the  institutional  level,  the  PFI  program  aims  to 
catalyze  innovation  in  smaller  institutions  and  in  those  that  do  not 
have  extensive  experience  performing  government-supported  re¬ 
search. 

At  the  project  level,  participants  suggested  that  individual  partner¬ 
ships  should  report  what  types  of  people  were  involved  in  the  part¬ 
nership  and  provide  basic  demographic  information  related  to  gen¬ 
der,  race  and  ethnicity,  and  other  factors. 

At  the  program  level,  achievement  of  diversity  goals  can  be  aggre¬ 
gated  up  from  project-level  data,  e.g.,  indicating  the  number  of  mi¬ 
nority,  female,  or  other  individuals  who  participated  in  all  of  the  PFI 
programs. 

Signposts.  The  signposts  were  essentially  identical  to  the  outcome 
measures. 
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CONCLUSIONS 

There  was  broad  support  in  the  workshop  for  a  number  of  proposi¬ 
tions  regarding  partnerships  and  networks.  First,  most  seemed  to 
believe  that  the  NSF  should  foster  technology  and  economic  devel¬ 
opment  through  local  networks  and  support  alternative  models  of 
partnerships,  clusters,  and  networks  for  encouraging  innovation. 
There  also  seemed  to  be  a  shared  view  that  the  PFI's  (and  NSF's) 
support  for  innovation  should  not  amount  to  an  entitlement.  The 
aim,  consistent  with  the  PFTs  program  goals,  should  be  to  catalyze 
innovation  by  bootstrapping  innovative  partnerships  and  networks 
and  providing  the  necessary  resources  and  technical  assistance  to 
improve  the  chances  that  they  could  become  self-sustaining.  It  was 
recognized,  however,  that  some  partnerships  would  achieve  sustain¬ 
ability  and  others  would  not.33 

Although  universities  were  seen  to  have  a  major  role  to  play  in  inno¬ 
vative  partnerships,  there  was  broad  support  for  the  proposition  that 
cultural  changes  are  needed  to  secure  that  role  and  to  ensure  balance 
between  educational,  research,  and  innovation  goals.  A  recurring 
complaint  was  that  university  policies  and  promotion  and  other 
reward  systems  too  often  fail  to  encourage — or  even  actively 
discourage — efforts  to  apply  and  commercialize  research.  Many 
workshop  participants  argued  that  this  needed  to  change,  even  as 
they  recognized  the  risks  to  educational  and  academic  research  mis¬ 
sions  of  becoming  overly  focused  on  commercialization  of  new 
knowledge. 


330ne  reviewer  of  this  report  suggested  that  sustainability  is  not  and  does  not  always 
have  to  be  a  goal  in  partnerships.  Indeed,  the  world  is  full  of  fluid  organizational  ar¬ 
rangements  to  drive  the  innovation  process.  However,  in  the  case  of  PFI,  sustainabil¬ 
ity  is  an  important  factor. 


Chapter  Five 

IMPLICATIONS  FOR  THE  NSF 


In  this  final  chapter,  we  summarize  the  main  conclusions  of  the 
workshop  regarding  the  evaluation  of  the  PFI  program  and  identify 
major  issues  related  to  the  NSF's  role  in  the  innovation  process.  We 
close  with  our  own  observations  regarding  the  sorts  of  questions  that 
the  NSF  should  seek  to  address  as  it  adapts  itself  to  the  changing 
needs  of  the  nation  in  the  new  century. 

EVALUATING  THE  PFI  PROGRAM 

Chapters  Three  and  Four  identified  a  large  number  of  suggested 
measures  related  to  various  issues  attendant  to  innovation  and  part¬ 
nerships.  We  now  step  back  from  these  details  to  summarize  the  ar¬ 
guments  related  to  the  broader  question  of  evaluating  the  PFI 
program. 

There  was  nearly  unanimous  support  for  a  formal  evaluation  of  the 
PFI  program  by  an  independent,  paid  evaluator.  Such  an  effort  was 
viewed  as  being  consistent  both  with  the  NSF's  general  commitment 
to  evaluation  as  an  aid  to  outcome-based  management  and  with  its 
specific  obligations  under  the  Government  Performance  and  Results 
Act  of  1993  (GPRA).1 

According  to  participants,  such  an  evaluation  should  rely  on  objec¬ 
tive  measures  that  can  lead  to  insights  regarding  the  most  critical 
questions  related  to  innovation  and  the  partnerships  themselves.  It 


^he  text  of  the  act  may  be  found  at  http://www.nsf.gov/od/gpra/law.htm. 
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also  should  provide  outcome  and  process  measures  (or,  as  used  in 
this  report,  signposts)  both  for  individual  projects  and  for  the  pro¬ 
gram  as  a  whole.2  Although  Pis  should  be  consulted  in  the  develop¬ 
ment  of  evaluation  metrics — and  should  assist  in  identifying  which 
are  appropriate  for  their  partnership — an  outside  evaluator  was 
viewed  as  necessary  to  ensure  objectivity  and  impartiality  in  the 
evaluation. 

As  described  above,  an  evaluation  of  the  PFI  program  also  should 
leverage  off  lessons  learned  and  best  practices  from  the  many  other 
NSF  programs  that  promote  innovation  and  link  universities  and  in¬ 
dustry.  Moreover,  many  argued  that  the  NSF  should  use  the  oppor¬ 
tunity  of  an  evaluation  to  turn  the  lens  back  on  itself:  To  inform  fur¬ 
ther  refinements  of  the  PFI  program,  the  evaluation  should  include 
measures  of  the  quality  of  the  technical  and  other  assistance  the  NSF 
is  providing  to  PFI  partnerships,  the  extent  to  which  that  support 
helped  the  partnerships,  and  what  other  assistance  also  might  be 
made  available. 

Although  some  seemed  to  imply  that  there  was  something  of  a  sense 
of  urgency  on  the  matter  of  an  evaluation — e.g.,  that  data  that  would 
be  needed  for  an  evaluation  might  not  be  collected  or  might  become 
lost— others  seemed  to  argue  that  since  actual  outcomes  could  not 
be  known  for  several  years  at  the  very  earliest,  this  was  not  a  particu¬ 
larly  urgent  matter.  In  either  case,  however,  the  basic  need  for  a 
proper  evaluation  of  the  PFI  program  was  generally  accepted. 

THE  NSF'S  ROLES  IN  INNOVATION 

As  described  in  the  preceding  chapters,  beyond  its  traditional  roles  of 
promoting  the  public  good  by  supporting  research  and  education, 
the  NSF  was  seen  by  participants  to  have  a  number  of  critical  roles, 
including  catalyzing  change,  forging  connections,  enhancing  institu¬ 
tions,  establishing  standards  and  identifying  best  practices,  enhanc¬ 
ing  diversity,  and  nurturing  alternative  models.  Each  will  be  sum¬ 
marized  next. 


2See  Appendix  G,  which  summarizes  workshop  discussions  regarding  both  some  de¬ 
sign  principles  and  the  potential  challenges  in  such  an  evaluation. 
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Catalyzing  Change 

The  NSF  has  a  unique  ability  to  lead  in  catalyzing  needed 
institutional  changes  in  the  university.  This  can  be  accomplished  by 
providing  an  NSF  stamp  of  approval  for  university  researchers 
working  with  industry  and  developing  ranking  criteria,  awards  and 
other  reward  systems,  and  other  incentives  that  can  foster  needed 
cultural  changes  in  universities  and  lead  to  greater  university  support 
for  the  application  of  research. 

Enhancing  Institutions 

One  means  for  catalyzing  change  is  creating  incentives  for  the  devel¬ 
opment  of  institutional  capacity  for  innovation.  This  can  include 
seeding  the  creation  of  multidisciplinary  or  industry-oriented  uni¬ 
versity-based  centers  of  excellence  and  assisting  existing  centers  in 
fostering  capacity  to  meet  industry  needs. 

Forging  Connections 

The  NSF  has  an  important  role  to  play  in  improving  the  performance 
of  the  information  markets  that  make  innovation  possible.  This  can 
be  accomplished  by  sponsoring  meetings  and  other  mechanisms 
that  heighten  the  prospects  that  interested  and  potentially  comple¬ 
mentary  parties  will  become  aware  of  one  another,  supporting  the 
development  of  networks  of  various  kinds  (e.g.,  alliances,  LISTSERVs 
with  threaded  discussions),  and  other  means. 

Establishing  Standards  and  Identifying  Best  Practices 

The  NSF  also  can  serve  as  an  honest  broker  in  developing  relevant 
standards  and  metrics,  identifying  lessons  learned  and  best  prac¬ 
tices,  and  ensuring  their  widest  possible  dissemination.  Leadership 
in  the  development  of  appropriate  standards  and  metrics  is 
needed — particularly  for  some  of  the  harder- to -measure  aspects  of 
innovation  and  partnerships — since  it  is  generally  beyond  the  ca¬ 
pacity  of  individual  partnerships  to  establish  a  consistent  framework 
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themselves.3  The  NSF's  diverse  portfolio  of  programs  that  focus  on 
innovation  and  fostering  university-industry  linkages  also  gives  it  a 
unique  capacity  to  compile  and  disseminate  lessons  learned  and  best 
practices  from  these  many  programs,  both  among  principal  investi¬ 
gators  and  among  broader  constituencies.4  The  NSF  also  can  spon¬ 
sor  conferences  of  awardees  who  meet  more  regularly  to  share 
insights,  as  well  as  supporting  other  means  of  enhancing  communi¬ 
cation  and  learning. 

Nurturing  Alternative  Models 

An  apparent  consensus  in  the  workshop  was  that  innovation  func¬ 
tionally  required  sufficiently  frequent  and  routine  interactions  be¬ 
tween  researchers  and  industry  to  develop  the  necessary  levels  of 
mutual  understanding  and  trust  that  could  lead  to  fruitful  partner¬ 
ships.  This  in  turn  required  stable  infrastructure  that  could  provide 
both  a  reliable  place  for  these  interactions  and  operational  support. 

Less  explicitly,  it  was  clear  from  the  workshop  that  participants  be¬ 
lieved  that  a  range  of  alternative  models  was  available  for  incubating 
and  otherwise  supporting  innovative  partnerships,  networks,  and 
other  arrangements  and  that  each  of  these  alternative  models  de¬ 
serves  support:  S&T  parks  and  campuses,  incubators  and  accelera¬ 
tors,  centers  of  excellence,  ERCs  and  I/UCRCs,  and  other  settings 
(see  the  box,  below). 

Indeed,  workshop  participants  seemed  to  endorse  both  an  expansion 
in  the  NSF's  support  for  innovation  and  partnerships  through  the  PFI 
and  continued  efforts  by  the  NSF  to  further  diversify  its  program¬ 
matic  support  for  innovation.  The  presentations  and  workshop  dis¬ 
cussions  evidenced  substantial  enthusiasm  for  the  PFI  program  and 
other  NSF  programs  that  support  innovation  and  university- industry 
collaborative  efforts,  without  favoring  any  particular  model  (e.g.,  PFI) 
over  any  other  (e.g.,  I/UCRCs).5 


3For  additional  reading,  see  Norling  (1997). 

4For  example,  some  suggested  that  the  NSF  might  arrange  for  speakers  from  the 
I/UCRC  program,  and  their  evaluators,  to  provide  their  insights  on  evaluation. 

5It  is  worth  pointing  out  that  a  number  of  workshop  attendees  were  neither  grantees 
nor  had  any  intention  of  applying  for  a  PFI  grant. 
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An  Infrastructure  Model  for  Innovation 

There  is  a  range  of  models  available  for  supporting  innovative  partnerships, 
networks,  and  other  arrangements.  Marye  Anne  Foxa  described  one  such 
model  being  used  at  North  Carolina  State  University. 

NC  State’s  Centennial  Campus  provides  physical  infrastructure  that  allows 
members  of  academe  and  industry  to  come  together  for  the  integration  of 
industry  activities  into  academic  programs,  and  brings  together  workforce, 
ideas,  new  approaches  to  education,  and  technology  transfer  on  a  single 
campus.  Centennial  Campus  is  a  carefully  planned  community  designed  to 
foster  innovation  and  support  a  unique  culture  of  partnerships  that  bring 
ideas,  people,  and  technology  together  in  new  ways.  It  is  unique  in  compari¬ 
son  to  other  types  of  research  parks,  in  that  it  looks  for  partnerships;  ways  for 
industry  or  government  partners  to  enhance  university  academic  and  edu¬ 
cational  programs  while  concurrently  bringing  value  to  what  they  do. 

The  goals  of  the  Centennial  Campus  have  been  to  break  down  some  of  the 
disciplinary  structures  that  have  been  developed  at  the  university  over  the 
last  few  years,  and  also  to  expand  the  results  to  industry.  The  campus  plan 
centers  around  its  R&D  neighborhoods,  which  are  each  dedicated  to  a  sub¬ 
ject  area  and  house  one  or  more  related  core  university  programs.  Buildings 
with  space  for  R&D  facilities  are  available  for  select  businesses  and  govern¬ 
ment  agencies  whose  operations  relate  to  these  program  areas.  The  million 
square  feet  of  physical  space  on  the  campus  has  been  generated  by  invest¬ 
ments  from  the  state,  private  sector,  and  from  government.  On  campus, 
there  are  53  companies  (16  large,  10  small,  and  27  startups),  seven  govern¬ 
ment  agencies,  seven  nonprofit  companies,  and  23  R&D  centers,  primarily 
funded  by  the  state.  900  employees  from  the  private  sector,  900  faculty,  staff 
and  post-docs,  and  1,400  students  are  involved  in  the  Campus.  Creation  of 
this  technology  infrastructure  permits  NC  State's  students  to  make  non- 
traditional  connections  with  industry  and  government. 

The  Centennial  Campus  is  just  one  model  available  for  supporting  and 
sustaining  innovation.  Other  models  include  traditional  S&T  parks  and 
campuses;  incubators  and  accelerators;  centers  of  excellence;  ERCs  and 
I/UCRCs;  and  other  settings. 


aFox  (2001). 
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The  workshop  devoted  less  attention  to  the  topic  of  selection  criteria 
and  the  difficulties  of  identifying  for  support  only  those  enterprises 
that  would  not  ordinarily  be  supported  by  industry.6 

LOOKING  TO  THE  FUTURE 

In  closing,  we  now  shift  from  one  role — faithful  reporters  of  the 
workshop  proceedings — to  another — policy  analysts  and  advisors  to 
the  NSF  and  the  PFI  program. 

As  the  NSF  takes  stock  of  its  past  experience  and  reflects  on  how  it 
will  express  its  future  commitment  to  catalyzing  innovation,  we  be¬ 
lieve  that  it  should  ask  itself  a  number  of  questions,  strategic  in  na¬ 
ture,  to  illuminate  the  benefits  and  costs  associated  with  different 
paths.  Among  these  are  the  following: 

•  Which  of  the  available  programs  and  models  for  promoting  in¬ 
novation  is  the  most  appropriate  tool  for  the  NSF  to  use  under 
which  circumstances? 

•  How  can  the  NSF  refine  its  understanding  of  the  relative 
effectiveness  of  its  innovation  programs  in  promoting  its 
objectives? 

•  In  what  ways  can  the  NSF  exploit  synergies  between  programs 
(e.g.,  PFI  programs  that  are  incubated  in  an  ERC  or  I/UCRC)? 

•  In  what  ways  can  (or  should)  this  mosaic  of  programs  and  mod¬ 
els  be  considered  together  as  a  larger  whole  to  ensure  that  the 
NSF’s  enterprise-wide  portfolio  of  innovation-catalyzing 
programs  matches  its  strategic  intent  and  its  presumed  desire  to 
achieve  an  optimal  program  mix  and  level  of  diversification? 

•  What  is  the  best  balance  or  tradeoff  among  the  various  poten¬ 
tially  conflicting  imperatives  (e.g.,  education  and  workforce  de¬ 
velopment,  academic  research,  innovation,  and  diversity)? 


6For  example,  one  would  want  to  make  sure  that  the  NSF  funds  do  not  “crowd  out” 
private  funding.  On  this  issue,  see  Wallsten  (2000),  and  David  and  Hall,  (2000). 
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•  What  factors  are  associated  with  the  success  or  failure  of  tech- 
nology  partnerships,  networks,  and  clusters?7 

Insofar  as  these  questions  seems  to  be  at  the  heart  of  the  NSF’s  effort 
to  build  a  new  foundation  for  innovation  while  remaining  true  to  its 
purpose,  they  seem  particularly  deserving  of  further  analysis,  dis¬ 
cussion,  and  debate. 


7On  technology  clusters,  see,  for  example,  Porter  (1990),  Sternberg  (1990),  Saxenian 
(1996),  BankBoston  Economics  Department  (1997),  Malecki  (1997),  Luger  (1999),  and 
Schmandt  (1999). 
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PFI  STEERING  GROUP  MEMBERS 


Table  A.1 

Workshop  Steering  Committee 


Name 

Institution 

William  Sibleya 

Oklahoma  Center  for  the  Advancement  of  Science  and 
Technology 

Hans  Brisch 

State  Higher  Education  Executive  Officers  Association 

Arturo  Bronson 

University  of  Texas  at  El  Paso 

Don  Cotten 

University  of  Southern  Mississippi 

Kerry  Davidson 

Louisiana  Board  of  Regents 

Larry  Farrar 

Montec  Research 

Jim  Lovelace 

University  of  Maryland 

Charles  Moreland 

North  Carolina  State  University 

Walter  Plosila 

Battelle  Memorial  Institute 

Ann  Redelfs 

University  of  California,  San  Diego 

aChairman. 
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PFI  GRANTEES 


Table  B.l 

FY  2001  Partnership  for  Innovation  Grantees 


Institution  (State) _ 

Arizona  State  University  (AZ) 

California  Institute  of  Technology 

Fisk  University  (TN) 

Ilisagvik  College  (AK) 

Indiana  University/Purdue 
University — Indianapolis  (IN) 

Morgan  State  University  (MD) 

North  Carolina  A&T  State  Uni¬ 
versity  (NC) 

Pennsylvania  State  University 
(PA) 

Rochester  Institute  of  Technology 
(NY) 

Rutgers  University  New 
Brunswick  (NJ) 

South  Dakota  State  University  (SD) 

Tennessee  Technological  Uni¬ 
versity  (TN) 

Texas  A&M  University  (TX) 

Tuskegee  University  (AL) 


_ _ Project _ 

AzPATH — A  Partnership  for  Housing  Innovation 
in  Arizona 

Entrepreneurial  Fellows  Program 
Room  temperature  infrared  lasers  based  on  rare 
earth  doped  CaGa2Se4 

Distance  Education  Delivery  for  Isolated  Rural 
Communities:  A  Contingency  Approach 
Partnerships  for  Innovation:  A  Center  of  Excel¬ 
lence  in  Regenerative  Biology 
Maryland  Technology  Partnership  for  Innova¬ 
tion 

Low  Cost  Resin  Transfer  Molded  Based 
Carbon/Carbon  Composites  for  Automotive  to 
Space  Applications 

A  Partnership  for  Innovation:  Promoting  Educa¬ 
tion  and  Research  in  Nanofabrication  Applica¬ 
tions  to  Biology  and  Medicine 
Upstate  Alliance  for  Innovation 

Models  for  Better  Academic-Industrial  Partner¬ 
ships  to  Create  Value  from  Concepts 
Great  Plains  Rapid  Prototyping  Consortium 
Expanding  Innovation  Opportunities  in  Ten¬ 
nessee 

Synergistic  Electronic  Commerce  (SynreCom) 
Partnership  for  Innovation 
A  Partnership  for  Innovations  in  Nancomposites 
Technology 
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Table  B.l  (continued) 


Institution  (State) 


Project 


University  of  Arkansas  (AR) 

University  of  Central  Oklahoma 
(OK) 

University  of  Idaho  (ID) 
University  of  Massachusetts- 
Amherst  (MA) 

University  of  Missouri-Kansas 
City  (MO) 

University  of  North  Carolina  Sys¬ 
tem  (NC) 


University  of  Puerto  Rico  (PR) 

University  of  Texas — Pan 
American  (TX) 

Virginia  Polytechnic  Institute  and 
State  University  (VA) 

West  Virginia  University  (WV) 


Innovation  Incubator:  Flaming  the  Sparks  of 
Creativity 

Institute  for  Emerging  Technologies:  Strategic 
Technology  Education  for  Non-Tech  Majors 

Farm  and  Ornamental  Fish 

Innovation  Networks:  A  Strategy  of  the  Regional 
Technology  Alliance 

The  Kansas  City  Regional  Innovation  Alliance 

North  Carolina  Technology  Development  Initia¬ 
tive:  A  Novel  Approach  to  Assess,  Disseminate 
and  Test  a  University /Venture  Capital/ 
Incubator  Partnership  Model 

Partnership  for  Innovation  to  Enhance  Puerto 
Rico's  Economic  Development 

Rapid  Product  Development  in  International 
Production 

Advanced  Materials  for  PEM-based  Fuel  Cell  Sys¬ 
tems 

Advanced  Polymer  Materials  for  Construction 
and  Aquaculture  Marketing  Development _ 


SOURCE:  National  Science  Foundation  Partnerships  for  Innovation  program. 
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PFI  WORKSHOP  ATTENDEES 


Table  C.l 

PFI  Workshop  Attendees 


Daniel  Akins,  City  College  of  New 
York 

Jeff  Alexander,  Washington  CORE  LLC 
Stuart  Arnett,  New  Hampshire 
Division  of  Economic  Development 
Phyllis  Arnette,  Texas  Instruments 
Incorporated 

Richard  Ash,  Mentor  Technology 
Ventures  LLC 

Philip  Auerswald,  Harvard  University 
Marietta  Baba,  Wayne  State 
University 

Gary  Bachula,  Internet 
Diola  Bagayoko,  Southern  University 
and  A&M  College 
Allen  Baker,  Vital  Strategies 
Anita  Balachandra,  Maryland 
Technology  Development 
Corporation 
Ramesh  Bapat,  RAND 
Howard  Bashford3,  Arizona  State 
University 

Richard  Bendis,  Kansas  Technology 
Enterprise  Corporation 
David  Benfield,  South  Dakota  State 
University 

William  Bernard,  TCE 
Joseph  Bordogna,  National  Science 
Foundation 


Donald  Boyda,  University  of 
Rochester 

Karen  Boykin,  University  of  Alabama 
Irene  Brahmakulam,  RAND 
Ernest  Brannon,  University  of  Idaho 
Michael  Breton3,  Rutgers  University 
LeeRoy  Bronner3,  Morgan  State 
University 

Arturo  Bronson,  University  of  Texas  at 
El  Paso 

Arnold  Burger3,  Fisk  University 
Chris  Busch,  University  of  Wyoming 
Research  Office 

Frederick  Byron,  Jr.3,  University  of 
Massachusetts 
Elias  Carayannis,  The  George 
Washington  University 
Michael  Champness,  Business-Higher 
Education  Forum 
Jaymie  Chernoff3,  University  of 
Massachusetts 
Mark  Coburn3,  University  of 
Rochester 

Julie  Cole,  Wake  Forest  University 
Ron  Cooper,  Center  for  Excellence  in 
IT  &  Telecommunications, 
Oklahoma  State  University 
Donald  Cotton,  University  of 
Southern  Mississippi 
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Ronald  Crawford,  University  of  Idaho 
Gerard  Crawley,  University  of  South 
Carolina 

Kenneth  Currie3,  Tennessee 
Technological  University 
George  Daddis,  Jr.a,  InScitek 
Microsystems,  Inc. 

Julio  Davalos3,  West  Virginia 
University 

Kerry  Davidson,  Louisiana  Board  of 
Regents 

Isadore  Davis,  Raytheon  Defense 
Systems 

Laura  DeNinno,  IT  Cluster/ 
Information  Technology 
Association  of  Southern  Arizona 
Jere  Denton,  CEO  Focus,  LLC 
Lee  Eiden,  U.S.  Department  of 
Education 

Barbara  Evans,  Lousiana  Partnership 
for  Technology  and  Innovation 
Larry  Farrar,  Montec  Research 
Stephen  Fonash3,  Pennsylvania  State 
University 

Mary  Anne  Fox,  North  Carolina  State 
University 

Kevin  Franklin,  University  of 
California,  San  Diego 
William  Gardiner,  Prince  George's 
Economic  Development 
Corporation 

Preston  Gilbert,  Syracuse  University 
Kinsey  Gimbel,  QRC  Division  of 
Macro  International,  Inc. 

Manuel  Gomez3,  University  of  Puerto 
Rico 

Miguel  Gonzalez3,  University  of 
Texas-Pan  American 
Mary  Good,  University  of  Arkansas, 
Little  Rock 

James  Gosz,  New  Mexico  EPSCoR, 
University  of  New  Mexico 
James  Green3,  Cumberland  Emerging 
Technologies 

Margaret  Grucza,  Industrial  Research 
Institute 

Joseph  Hammang,  Rhode  Island 
Economic  Policy  Council 


Marion  Harmon,  Florida  A&M 
University 

David  Harris3,  University  of  Central 
Oklahoma 

Steve  Hart,  Interpretech,  LLC 
Charles  Hatch3,  University  of  Idaho 
Robert  Heard,  National  Association  of 
Seed  and  Venture  Funds 
Christopher  Hill,  George  Mason 
University 

Chris  Hollinsed,  DuPont  Central 
Research  and  Development 
Gary  Holloway3,  Evendale  Bearings, 
Seals,  and  Drives 
Kent  Hughes,  Woodrow  Wilson 
International  Center  for  Scholars 
John  Hurt,  National  Science 
Foundation 
Brian  Jackson,  RAND 
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JUNE  18,  2001 

8:25  Welcome 

—  Steven  Popper,  Senior  Economist,  RAND 

8:30  Opening  Remarks 

—Joseph  Bordogna,  Deputy  Director,  National  Science 
Foundation 

8:50  Overview  of  the  Partnerships  for  Innovation  (PFI)  Program 

—John  Hurt,  Program  Director,  PFI,  National  Science  Foun¬ 
dation,  and  Mariann  (Sam)  Jelinek,  Program  Director,  In¬ 
novation  and  Organizational  Change,  National  Science 
Foundation 

9:15  An  Industry  Perspective  on  Partnerships 

— Egils  Milbergs,  President,  National  Coalition  for  Advanced 
Manufacturing 

9:45  A  University  Perspective  on  Partnerships 

— Marye  Anne  Fox,  Chancellor,  North  Carolina  State  Uni¬ 
versity 

10:15  Break 

10:30  Panel  Discussion:  Partnerships,  Clusters,  and  Networks 
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-Moderator:  J.  David  Roessner,  Associate  Director,  Science 
and  Technology  Program,  SRI  International 

1.  Local  and  Regional  View— Richard  Bendis,  President, 
KTEC 

2.  National  View-Kathleen  (Taffy)  Kingscott,  Director, 
Public  Affairs,  IBM 

3.  International  View— Mary  Good,  Dean,  Donaghey 
College  of  Information  and  Systems  Engineering, 
University  of  Arkansas,  Little  Rock 

1 1 :30  Question-and-Answer  Session 

12:00  Lunch 

-The  Importance  of  Sustainability  and  Ways  to  Achieve  It, 
Kent  Hughes,  Public  Policy  Scholar,  Woodrow  Wilson  In¬ 
ternational  Center  for  Scholars 

1:15  Breakout  Sessions-two  sessions  each  per  topic: 

1 .  From  Research  to  Commercialization  ( 1 ) 

Resource:  Gary  Holloway,  Evendale  Bearings,  Seals 
and  Drives;  Kunigal  Shivakumar,  North  Carolina 
A&T  State  University 

Facilitator:  Larry  Farrar,  Montec  Research 

2.  From  Research  to  Commercialization  (2) 

Resource:  Frederick  Byron,  Jr.,  and  Jaymie  Chernoff, 
University  of  Massachusetts  Amherst;  Jerome  Schau- 
feld,  Mass  Ventures  Corporation 

Facilitator:  Don  Cotten,  University  of  Southern  Mis¬ 
sissippi 

3.  Infrastructure  Development  for  Research  and  Com¬ 
mercialization  (1) 
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Resource:  Ken  Currie,  Tennessee  Technological 
University;  James  Green,  Cumberland  Emerging 
Technologies 

Facilitator:  William  Sibley,  Oklahoma  Center  for  the 
Advancement  of  Science  and  Technology 

4.  Infrastructure  Development  for  Research  and  Com¬ 
mercialization  (2) 

Resource:  David  Winwood,  University  of  North 
Carolina;  J.  Ted  Morris,  NC  Technological  Develop¬ 
ment  Authority,  Inc. 

Facilitator:  Charles  Moreland,  North  Carolina  State 
University 

5.  Workforce  Development  and  Enhancement  (1) 

Resource:  Stephen  Fonash,  Pennsylvania  State  Uni¬ 
versity 

Facilitator:  John  Hurt,  National  Science  Foundation 

6.  Workforce  Development  and  Enhancement  (2) 

Resource:  David  Harris,  University  of  Central  Okla¬ 
homa;  George  Reeder,  TVR  Communications 

Facilitator:  AnnRedelfs,  University  of  California,  San 
Diego 

7 .  Intellectual  Property  Concerns  and  Barriers  (1) 1 

Resource:  Manuel  Gomez,  University  of  Puerto 
Rico;  Victor  Rivera,  University  of  Puerto  Rico;  Abe 
Schwartz,  Caribbean  MicroParticles  Corporation 

Facilitator:  Brian  Jackson,  RAND 


l 


Consolidated  into  a 


single  session. 
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Intellectual  Property  Concerns  and  Barriers  (2) 

Resource:  Joachim  Kohn,  Rutgers  University;  Salva¬ 
tore  Romano,  New  Jersey  Center  for  Biomaterials 

Facilitator:  Bruce  McWilliams,  The  George  Washing¬ 
ton  University 


3:30  Break 

3:45  Reports  from  Breakout  Sessions 

5:00  Partnerships  for  Innovation  Awardees  Poster  Session/ Social 

JUNE  19, 2001 

8:00  Venture  Capital  for  Partnerships 

-E.  Roger  Novak,  Jr.,  Founding  Partner,  Novak  Biddle  Ven- 
turePartners 

8:30  Assessment  and  Performance  Measures 

— Maryellen  Kelley,  Partner,  Pamet  Hill  Associates 
9:00  Breakout  Sessions— two  sessions  per  topic: 

1.  Sustainability  (1) 

Resource:  Isadore  Davis,  Raytheon  Defense  Sys¬ 
tems;  Laura  DeNinno,  IT  Cluster/ITASA 

Facilitator:  Kerry  Davidson,  Louisiana  Board  of  Re¬ 
gents 

2.  Sustainability  (2) 

Resource:  Donald  Boyd,  Mark  Coburn,  and  Marjorie 
Zack,  University  of  Rochester;  George  Daddis, 
InSciTek  Microsystems,  Inc. 


^Consolidated  into  a  single  session. 
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Facilitator:  Walter  Plosila,  Battelle  Memorial  Insti¬ 
tute 

3.  Metrics /Evaluation /Assessment  (1) 

Resource:  Mark  Morgan,  QRC  Division  of  Macro 
International,  Inc. 

Facilitator:  Julio  Lopez-Ferrao,  National  Science 
Foundation 

4.  Metrics /Evaluation /Assessment  (2) 

Resource:  Christopher  Hill,  George  Mason  Univer¬ 
sity 

Facilitator:  Jonathon  Tucker,  George  Mason  Uni¬ 
versity 

5.  Management  and  Financial  Planning  (1) 

Resource:  LeeRoy  Bronner,  Morgan  State  University; 
Philip  Singerman,  Maryland  Technology  Develop¬ 
ment  Corporation 

Facilitator:  Arturo  Bronson,  University  of  Texas  at  El 
Paso 

6.  Management  and  Financial  Planning  (2) 

Resource:  Robert  Heard,  National  Association  of 
Seed  and  Venture  Funds 

Facilitator:  Philip  Auerswald,  Harvard  University 

11:15  Break 

1 1 :30  Reports  from  Breakout  Sessions 

Wrap-Up  and  Concluding  Remarks:  John  Hurt,  National  Sci¬ 
ence  Foundation 


12:15 


Appendix  E 

CHARGES  TO  BREAKOUT  SESSIONS 


This  appendix  documents  the  questions  each  breakout  session  was 

charged  with  addressing. 

JUNE  18  SESSIONS 

From  Research  to  Commercialization 

•  What  are  the  main  characteristics  of  commercially  useful  re¬ 
search? 

•  What  are  the  most  efficient  ways  for  the  academic  sector  to  target 
firms  that  might  want  to  partner  with  them  and  can  provide  the 
needed  resources? 

•  What  are  the  most  efficient  ways  for  firms  to  find  academic  part¬ 
ners? 

•  What  are  the  principal  challenges  or  barriers  to  commercializa¬ 
tion? 

•  What  are  the  best  means  for  eliminating  them  or  reducing  their 
effect? 

•  What  roles  can  the  NSF  play  in  the  commercialization  process? 

Infrastructure  Development  for  Research  and 

Commercialization 

•  What  are  the  principal  types  of  infrastructure  needed 
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—  To  foster  the  commercialization  of  research? 

—  To  communicate  to  potential  users  the  goals  of  research,  its 
relevance  to  their  commercial  products  and  processes? 

—  To  disseminate  research  results  to  potential  users? 

—  To  scale  up  to  commercial  application  (e.g.,  financial  capital, 
learning  curves)? 

—  To  fully  exploit  the  value  of  patents  and  other  IP? 

•  What  roles  can  the  NSF  play  in  fostering  the  development  of 
needed  infrastructure? 

Workforce  Development  and  Enhancement 

•  What  are  the  principal  types  of  infrastructure  needed 

—  To  foster  workforce  development  and  enhancement? 

—  To  educate  and  train  human  capital  for  the  research  enter¬ 
prise? 

—  To  educate  and  train  the  workforce? 

•  How,  if  at  all,  does  the  NSF  need  to  change  or  improve  upon  its 
role  in  workforce  development  and  enhancement? 

Intellectual  Property  Concerns  and  Barriers 

•  What  types  of  benefits  (e.g.,  revenues  from  licensing  and  patents, 
sponsored  research,  internships  for  students,  funding  for  equip¬ 
ment)  can  researchers  derive  from  their  IP? 

•  What  are  the  principal  IP  issues  that  need  to  be  successfully  re¬ 
solved  by  each  of  the  partnerships? 

—  From  the  university  perspective 

—  From  the  industry  perspective 

•  What  are  the  main  challenges  or  barriers  to  their  resolution? 

•  What  are  the  best  means  for  resolving  disagreement? 
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•  What  if  any  roles  can  the  NSF  play  in  assisting  in  resolving  IP 
concerns  and  barriers? 

JUNE  19  SESSIONS 

Sustainability 

•  What  are  likely  to  be  the  key  characteristics  of  sustainable  part¬ 
nerships? 

•  What  means  are  available  for  developing  these  characteristics? 

•  What  are  the  key  challenges  or  barriers  to  their  emergence  and 
how  can  they  be  overcome? 

•  What  if  any  roles  can  the  NSF  play  in  reducing  or  eliminating 
these  challenges  and  barriers? 

Metrics/Evaluation/Assessment 

•  What  process  measures  should  be  used  to  evaluate  progress  be¬ 
fore  actual  outcomes  emerge  from  each  partnership? 

•  What  outcome  measures  should  be  used  to  evaluate  results  at 
the  end  of  each  partnership? 

•  What  other  performance  measures  should  be  used  in  assessing 
the  partnerships? 

Management  and  Financial  Planning 

•  What  are  the  key  characteristics  of  a  successfully  managed  part¬ 
nership? 

•  What  are  the  main  challenges  or  barriers  to  successful  manage¬ 
ment  and  what  means  are  available  for  overcoming  them? 

•  What  are  the  key  elements  of  a  good  financial  plan? 

•  What  are  the  main  challenges  or  barriers  to  financial  planning 
and  what  means  are  available  for  overcoming  them? 

•  What  roles  can  the  NSF  play  that  would  be  both  useful  and 
appropriate? 


Appendix  F 

SUMMARY  OF  KEY  WORKSHOP  FINDINGS  AND 

RECOMMENDATIONS 


The  NSF  has  developed  a  set  of  goals  to  complement  its  overall 
vision:  (1)  people,  (2),  ideas,  and  (3)  tools.  Most  of  the  comments 
and  recommendations  from  the  breakout  sessions  have  been 
categorized  in  Table  F.l  under  these  three  main  headings.  The 
remaining  points  were  categorized  under  sustainability,  which 
transcends  and  pertains  to  these  three  headings. 

Workshop  participants  identified  two  main  areas  of  focus  within  the 
people  category:  current  and  future  workforce,  in  general,  and  those 
specifically  involved  in  partnerships.  Participants  also  recognized 
ideas  leading  to  new  knowledge  power  innovation  and  productivity 
in  today’s  economy.  Ideas  must  be  communicated  and  shared 
between  university  and  industry.  Each  must  gain  an  understanding 
of  the  other’s  capabilities  and  expertise.  Communication  helps  to 
bridge  gaps  and  aids  new  knowledge  gained  from  research  to  be 
commercialized  (the  process  of  innovation). 

Finally,  it  was  realized  that  tools  are  needed  to  advance  the  frontiers 
in  every  field.  These  tools,  as  identified  by  workshop  participants, 
come  in  the  form  of  (1)  infrastructure,  (2)  intellectual  property,  and 
(3)  metrics,  evaluation,  and  assessment.1 


Process  measures  should  be  developed  keeping  NSF  internal  reporting  requirements 
in  mind. 
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For  that  type  of  effort,  a  wants  to  grow,  build,  and  Character  of  innovation 
plan  that  will  ensure  the  sustain  itself.  matters, 

continuation  and  liveli-  Lack  of  funding  for  op  -  Incremental  improvements 
hood  of  a  partnership  erational  support,  PR,  can  ensure  future, 

should  be  developed. 


Appendix  G 

OBSERVATIONS  ON  PFI  EVALUATION  CRITERIA 


In  light  of  the  NSF's  commitment  to  measuring  performance  and  re¬ 
sults,  there  was  strong  support  for  undertaking  a  proper  evaluation 
of  the  PFI  program.  The  preceding  chapters  reported  on  a  number  of 
specific  outcome  measures  and  process  measures  (signposts)  that 
were  suggested  to  assess  program  performance;  we  now  turn  to 
some  of  the  broader  principles  that  were  articulated  (or  could  be  in¬ 
ferred)  from  the  workshop  discussions. 

SOME  GENERAL  PRINCIPLES  FOR  EVALUATION 

Underlying  beliefs  about  the  fundamental  nature  of  the  innovation 
process  and  the  characteristics  of  successful  partnerships  impor¬ 
tantly  shaped  views  regarding  appropriate  selection  and  assessment 
criteria  for  innovative  partnerships  and  the  sorts  of  foundations  that 
are  necessary  to  ensure  their  sustainability.  Participants  posited  a 
number  of  general  principles  that  might  guide  the  development  of 
evaluation  criteria  for  the  PFI  program: 

•  It  is  important  to  measure  both  innovation  and  partnerships. 

•  There  are  important  differences  between  program  evaluation  cri¬ 
teria,  i.e.,  criteria  by  which  the  benefits  and  costs  of  the  overall 
PFI  program  should  be  judged,  and  project  evaluation  criteria 
that  are  needed  to  generate  data  for  the  overall  program  assess¬ 
ment.  Program  evaluation  criteria  should  be  established  before 
project  evaluation  criteria  and  should  explicitly  tie  project-level 
measures  to  the  larger  program  evaluation. 


89 


90  Building  a  New  Foundation  for  Innovation:  Results  of  a  Workshop 


•  There  are  important  differences  between  outcome  measures  that 
would  be  measurable  only  at  the  end  of  a  partnership  and  process 
measures  that  could  be  used  while  the  project  was  under  way.1 
Outcome  measures  should  be  established  before  process  mea¬ 
sures. 

•  In  all  cases,  participants  favored  the  development  of  “objective” 
measures  of  evaluation,  recognizing  the  difficulties  in  developing 
such  measures. 

•  Benchmarking  and  establishing  indicators  as  a  standard  of  com¬ 
parison  need  to  be  done  at  the  beginning  of  a  project  to  establish 
a  continuous  data  flow  and  ensure  that  relevant  data  are  not  lost. 

•  Finally,  workshop  discussants  asked  who  (e.g.,  the  Pis  them¬ 
selves  or  independent  evaluators)  should  perform  the  outcome 
evaluations  and  when?  Although  it  is  important  that  Pis  provide 
input  on  the  performance  measures  that  will  be  used,  most  sup¬ 
ported  independent,  paid  evaluators. 


OUTCOME  MEASURES 

There  was  some  agreement  on  the  principles  that  should  guide  the 
development  of  metrics,  even  if  there  remained  a  number  of  unre¬ 
solved  questions: 

•  Many  favored  development  of  a  “strong  logic”  for  the  choice  of 
goals  and  metrics  to  assess  their  accomplishment,  while  provid¬ 
ing  the  flexibility  necessary  to  enable  application  to  a  wide  range 
of  projects.2 

•  The  PFI  program's  multiple  program  goals  has  led  to  diversity  in 
the  PFI  portfolio  of  projects,  which  also  promotes  multiple  pro¬ 
ject  goals  and  makes  it  difficult  to  make  comparisons.  Although 
discussants  agreed  that  not  every  project  needed  to  fulfill  each 


^For  example,  outcome  measures  were  viewed  as  end  results  and  summative  in  na¬ 
ture,  useful  for  assessing  goal  accomplishment  at  the  end  of  a  partnership,  whereas 
process  measures  were  seen  as  real-time  and  formative,  useful  for  providing  feedback 
for  improving  the  program. 

2The  metrics  used  should  depend  on  the  goals  and  the  emphasis  of  work  (e.g.,  work¬ 
force  vs.  technical  studies). 
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PFI  goal,  and  that  projects  should  be  evaluated  on  the  basis  of 
what  they  originally  proposed  to  do,  questions  remained  as  to 
whether  multiple  goals  should  be  equally  weighted  in  an  evalua¬ 
tion. 

•  Benchmarks  need  to  be  established,  even  if  those  benchmarks 
are  somewhat  imperfect.  Initially,  one  may  be  able  only  to  estab¬ 
lish  benchmarks  derived  from  the  goals  of  the  partnership  and 
then  to  ask  whether  (or  the  degree  to  which)  it  achieved  specific 
goals;  these  may  change  over  time,  but  this  provides  an  initial 
basis  for  comparison,  or  straw  man.3 

•  Some  respondents  argued  that  it  was  more  important  to  get  an¬ 
swers  to  fundamental  questions  rather  than  more  esoteric  ones 
that  might  need  to  rely  on  sophisticated  measurement  tech¬ 
niques.  For  example,  how  was  the  partnership  formed?  How 
was  the  venture  expanded?  What  were  the  specific  goals  to  be 
achieved?  Were  the  goals  achieved? 

In  navigating  these  various  imperatives  and  challenges,  workshop 
discussions  suggested  a  combination  of  top-down  and  bottom-up 
approaches  that  would  need  to  be  connected  to  perform  both  pro¬ 
gram-  and  project-level  outcome  assessments: 

•  On  the  one  hand,  participants  supported  a  top-down  approach 
in  which  program  evaluation  criteria  first  were  established  based 
upon  program  goals,  and  project-level  outcomes  mapped  to 
these  larger  program  goals; 

•  On  the  other  hand,  it  was  clear  that  judgments  about  the  out¬ 
come  of  the  overall  PFI  program  rested  upon  a  summation  of 
judgments  about  the  individual  projects,  and  the  heterogeneity 
of  the  projects  suggested  that  a  fair  amount  of  tailoring  might  be 
required. 

To  illuminate  some  of  these  interdependencies,  we  now  summarize 
views  regarding  the  top-down  program-level  outcome  measures  that 


3For  example,  a  partnership  might  have  the  goal  of  funding  12  graduate  students,  with 
seven  of  them  going  to  work  in  this  area  after  graduation  and  the  project  self-sustain¬ 
ing  in  three  years.  At  the  end  of  the  evaluation  period,  the  partnership  might  have 
fallen  short  of  its  goals,  but  the  initial  goals  would  provide  a  seemingly  reasonable  ini¬ 
tial  point  of  comparison. 
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derive  from  PFI  program  goals  and  the  sorts  of  summative  judgments 
that  will  need  to  be  made.  We  then  discuss  workshop  participants' 
views  on  project-level  outcome  measures  and  how  these  project- 
level  measures  might  be  summed  in  such  a  way  that  they  can  be  re¬ 
connected  to  the  program-level  evaluation. 

Program-Level  Outcome  Measures 

Workshop  participants  argued  that  the  PFI  program  should  largely 
be  evaluated  on  the  basis  of  whether  it  achieved  its  goals,  including 
stimulating  the  transformation  of  knowledge,  sustaining  innovation, 
and  transferring  technology;  training  and  workforce  development; 
catalyzing  economic  development  of  states  and  regions;  and  broad¬ 
ening  participation  in  the  innovation  enterprise. 

Leaving  aside  the  inherent  difficulties  of  developing  outcome  mea¬ 
sures  for  these  goals,4  there  also  was  some  support  for  evaluating  the 
program  on  the  basis  of  several  less  easily  measured  outcomes. 
Among  these  were  estimating  the  value  added  by  the  partnership, 
including  the  types  of  benefits,  number  of  beneficiaries,  and  distri¬ 
bution  of  these  benefits;  the  spread  of  ideas  on  how  to  partner  effec¬ 
tively  and  ensure  best  practices;  the  sustainability  and  long-term  ef¬ 
fect  of  the  partnerships;  the  amount  of  new  knowledge  gained;  and 
whether  the  award  increased  the  propensity  of  awardees  to  partner 
more  in  the  future. 

There  also  was  some  support  for  the  notion  that  the  outcome  mea¬ 
sures  should  be  capable  of  informing  decisions  about  how  to  im¬ 
prove  the  PFI  program:  to  learn  which  things  are  working,  identify 
bottlenecks  and  “failure  nuggets"  and  remove  them,  and  how  to 
better  perform  risk  analyses.  There  also  was  a  strong  sentiment  that 
such  evaluative  information  could  facilitate  the  identification  of  what 
the  most  effective  strategies  might  be  and  what  might  constitute  best 
practices  at  the  national  level. 


4One  challenge  in  evaluating  outcomes  is  the  inherent  difficulty  of  counterfactual 
analyses.  For  the  PFI  program,  one  needs  to  establish  that  the  partnership  would  not 
have  occurred  and  the  stream  of  benefits  that  resulted  from  the  partnership  would  not 
have  been  seen  had  the  PFI  grant  not  been  provided.  To  do  this,  one  would  have  to  in¬ 
terview  the  proposers  of  those  projects  that  were  not  funded  and  discover  whether 
those  partnerships  were  in  fact  established  even  without  PFI  funding. 
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Project-Level  Outcome  Measures 

Although  workshop  participants  felt  that  project  evaluation  metrics 
should  be  tied  to  the  PFI  program's  stated  goals,  they  also  recognized 
that  the  partnerships  were  of  many  different  types,  promoting  differ¬ 
ent  mixes  of  the  program  goals.  The  key  issues  regarding  outcome 
evaluations  at  the  project  level,  then,  were  determining  which  goals 
applied  to  which  project  and  what  combination  of  metrics  accord¬ 
ingly  should  be  used  to  evaluate  goal  accomplishment  for  each,  so 
that  these  results  could  provide  the  data  necessary  for  the  program 
assessment. 

Additionally,  some  expressed  the  view  that  at  the  project  level,  pro¬ 
jects  could  (and  should)  be  evaluated  largely  on  the  basis  of  whether 
they  had  met  the  specific  goals  detailed  in  the  original  proposal  to 
the  PFI.  To  this  end,  it  was  believed  that  principal  investigators 
should  initiate  conversations  with  the  NSF  over  which  criteria  would 
be  used  to  assess  which  outcomes  of  their  partnership.5 

PROCESS  MEASURES 
Program-Level  Signposts 

The  program-level  signposts  were  seen  as  summative,  based  upon 
project-level  measures,  described  next. 

Project-Level  Signposts 

As  was  described  in  preceding  appendixes,  attendees  advocated 
project-level  process  measures  (signposts)  of  two  general  kinds.  The 
first  kind  focused  on  the  progress  of  the  partnership  in  achieving  its 
innovation  goals  (which  could  indicate  whether  partnership  activi¬ 
ties  were  leading  toward  results);  the  second  focused  on  the  charac¬ 
teristics  of  the  partnership  that  could  indicate  whether  the  partner¬ 
ship  was  using  available  instruments  in  pursuit  of  the  correct  actions. 


5Although  the  idea  may  have  some  merit,  there  was  no  suggestion  that  PFI  applicants 
should  include  in  their  application  a  suggested  set  of  evaluation  measures  for  their 
project. 
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EVALUATING  THE  NSFS  OWN  SUPPORT 

Finally,  consistent  with  the  view  that  the  evaluation  process  pre¬ 
sented  the  NSF  with  an  opportunity  to  turn  the  lens  on  itself  in  an 
effort  to  improve  its  support  for  innovation,  attendees  suggested  that 
the  partnerships  should  be  asked  to  evaluate  the  NSF's  support  dur¬ 
ing  the  PFI  program  and  the  extent  to  which  that  support  helped 
their  partnership.  It  was  unclear  how  best  to  acquire  this  informa¬ 
tion  (e.g.,  by  open-ended  responses  or  a  by  standardized  rating 
scheme),  but  the  basic  aim  was  to  elicit  partners’  views  on  such 
questions  as  whether  the  program  had  been  a  success  and  worth¬ 
while  experiment  from  their  vantage  point,  whether  they  would  rec¬ 
ommend  the  program  to  others,  and  what  they  learned  about  boot¬ 
strapping  the  innovation  process.  The  project-level  evaluations  of 
the  value  of  the  NSF's  support  also  were  viewed  as  summative  and 
easily  aggregated  for  the  program  as  a  whole. 

Some  also  saw  program-level  process  measures  as  the  place  for  the 
NSF  to  turn  the  lens  on  itself  regarding  selection  and  grant-making 
criteria,  the  quality  of  the  technical  assistance  it  provides,  and  other 
issues.6  The  quality  of  the  technical  assistance  provided  by  the  NSF 
can  be  summed  from  the  individual  project-level  assessments  of  how 
appropriate  and  helpful  that  technical  assistance  was. 

Workshop  participants  identified  a  wide  range  of  challenges  related 
to  the  methodology  for  evaluating  individual  projects  and  the  pro¬ 
gram  as  a  whole  and  in  some  cases  offered  practical  suggestions  re¬ 
garding  how  best  to  implement  an  evaluation  of  the  PFI  program. 
We  now  consolidate  and  summarize  these  observations  and 
suggestions. 

•  The  Difficulties  of  Evaluation  in  Observational  Studies.  One 
challenge  arises  from  the  fact  that  a  PFI  evaluation  would  not  be 
a  controlled  experiment7  but  an  observational  study,  requiring 


6For  example,  the  time  it  takes  to  make  an  award  was  seen  as  important. 

7In  other  words,  projects  were  not  randomly  selected  to  receive  a  treatment  (funding), 
which  would  have  enabled  a  comparison  between  those  that  were  funded  and  those 
that  were  not. 
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careful  handling  of  potential  bias.8  To  evaluate  outcomes  in 
such  a  setting,  one  needs  to  compare  the  outcomes  of  the  chosen 
partnerships  with  some  referent,  but  it  was  somewhat  unclear 
exactly  what  that  referent  should  be.  Should  a  partnership's  ex¬ 
perience  be  compared  to  a  counterfactual  where  the  partnership 
did  not  receive  PFI  funding?  What  sort  of  confidence  could  one 
have  in  such  a  comparison?  If  partnerships  are  to  be  compared 
to  projects  whose  proposals  were  not  selected  by  the  PFI 
program,  how  should  the  inherent  biases  be  handled? 

•  Self-Evaluation  Versus  Independent  Evaluators.  There  was  sup¬ 
port  for  including  stakeholders  in  determining  how  their  projects 
should  be  measured,  but  there  was  somewhat  mixed  support  for 
the  broader  principle  of  self-evaluations.  On  the  one  hand,  some 
considered  the  prospect  of  self-evaluation  as  presenting  an  op¬ 
portunity  that  might  be  abused;  others  perceived  a  financial  in¬ 
volvement  that  lent  credence  to  self-evaluation  and  straightfor¬ 
ward  numbers  that  partners  can  provide.  In  light  of  the  concern 
that  self-reported  measures  could  be  biased,  there  was  broad 
support  for  bringing  in  an  external  evaluator;  the  point  also  was 
made  that  NSF  programs  require  a  paid,  outside  evaluator  for  all 
projects. 

•  Maintaining  Records.  Another  challenge  was  collecting  and 
maintaining  the  necessary  records  to  inform  the  evaluation;  it  is 
important  that  partners  establish  a  process  early  in  the  project 
for  collecting  and  maintaining  records  that  can  provide  the  NSF 
with  relevant  performance  information.  Some  argued  that  in¬ 
centives  might  be  established,  for  example,  tying  the  evaluation 
to  the  collection  of  information  that  also  could  be  used  later  in 
marketing  the  project.  For  others,  the  main  purpose  was  not  so 
much  to  evaluate  a  particular  project  but  to  evaluate  the  overall 
program.  This  information  was  seen  to  be  useful  in  coming  to 
conclusions  regarding  what  works  and  what  does  not  and  for 
supporting  recommendations  to  policymakers  so  that  program 
design  can  be  improved. 


8The  selected  partnerships  were  those  that  appeared  to  have  the  most  promising  pro¬ 
posals,  and  those  that  were  not  selected  had  proposals  that  were,  in  some  sense, 
judged  inferior. 
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•  Time  Horizon  for  Evaluation.  Some  thought  that  the  time  hori¬ 
zon  of  the  PFI  program  was  too  short  to  be  able  to  ascertain  the 
program’s  effect  on  innovation  and  sustainability.  For  them,  it 
was  just  too  short  a  time  scale  for  an  evaluation.  Some  suggested 
a  “3+2”  year  evaluation  period — three  years  for  the  initial 
support,  with  two  years  of  follow-up  on  outcomes.  By  the  same 
token,  a  two-to-three-year  time  horizon  was  seen  as  probably  be¬ 
ing  long  enough  to  evaluate  the  NSF’s  role  as  a  catalyst. 

•  Data  Reliability.  Several  workshop  participants  expressed  con¬ 
cern  about  the  comprehensiveness,  validity,  and  reliability  of 
performance  data.9  Observing  that  data  can  be  no  more 
accurate  than  the  way  it  is  reported,  it  is  important  to  make  data 
collection  as  simple  as  possible.  The  observation  was  made  that 
too  many  studies  hide  behind  a  great  deal  of  complexity,  and  it  is 
quite  difficult  to  establish  that  the  underlying  data  are  even  valid. 
Given  that  the  PFI  program  is  small,  it  should  be  relatively  easy  to 
interview  all  participants. 


9They  cited  the  case  of  the  SBIR  reauthorization  as  evidence  of  the  importance  of  reli¬ 
able  and  valid  data. 
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In  2000,  the  National  Science  Foundation  (NSF)  created  the 
Partnerships  for  Innovation  (PFI)  program.  The  PFI  is  part  of 
a  larger  NSF  effort  to  build  a  new  foundation  for  innovation 
based  upon  partnerships  between  university,  industry,  and 
local  and  regional  governments  that  also  will  be  responsive  to 
emerging  economic  and  social  challenges  facing  the  nation. 

In  June  2001,  a  workshop  held  in  Arlington,  Virginia, 
brought  together  PFI  grantees  and  representatives  from 
university  and  industry  to  consider  the  roles  of  the  PFI  and 
the  NSF  in  the  larger  national  innovation  enterprise.  This 
report  synthesizes  workshop  discussions  regarding  innovation 
and  sustainable  partnerships.  The  workshop  revealed  strong 
support  for  a  formal  evaluation  of  the  PFI  program  and 
endorsed  both  an  expanded  NSF  role  in  promoting 
innovation  and  partnerships  through  the  PFI  and  other 
programs  and  continued  efforts  by  NSF  to  further  diversify 
and  better  exploit  synergies  between  its  innovation- 
supporting  programs. 
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